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ABSTRACT 


We  have  developed,  tested,  and  operationally  implemented  a  web  based 
system  for  collecting  and  analyzing  in  near-real  time  weather  forecast  and 
observational  data  to  assess:  (a)  the  performance  of  forecasts;  and  (b)  the 
operational  impacts  of  forecasts.  A  major  goal  of  the  system  is  to  quantify  the 
impacts  of  weather  forecasts  on  the  planning,  execution,  and  outcomes  of 
military  operations.  Our  tests  and  implementation  were  focused  on  the  METOC 
support  provided  by  Naval  Pacific  Meteorology  and  Oceanography  Detachment 
(NPMOD)  Fallon  to  Naval  Strike  and  Air  Warfare  Center  (NSAWC)  operations  at 
Naval  Air  Station  Fallon.  In  this  application  of  the  system,  METOC  and  NSAWC 
data  are  collected  by  NPMOD  Fallon  personnel  and  entered  via  a  web  interface 
into  a  database  at  the  Naval  Postgraduate  School  (NPS)  where  the  data  are 
analyzed  and  results  are  reported  in  near-real  time.  The  results  include 
quantitative  assessments  of:  (1)  forecasts  used  in  planning  NSAWC  missions 
(e.g.,  forecast  accuracy,  probability  of  detection);  (2)  changes  made  during 
mission  planning  in  response  to  forecasted  weather  (e.g.,  changes  in  mission 
schedule,  targets,  weapons,  tactics);  (3)  deviations  from  mission  plans  that 
occurred  during  missions  in  response  to  weather  conditions  actually  encountered 
by  air  crews  (e.g.,  changes  in  targets,  weapons  delivered,  tactics);  (4)  positive 
and  negative  impacts  on  mission  planning,  execution,  and  outcomes  due  to 
forecasts  (e.g.,  missions  that  avoided  or  incurred  delays,  cancellations, 
inappropriate  weapons  load  outs,  missions  that  might  have  avoided  problems 
had  the  forecast  been  followed  by  mission  planners);  (5)  METOC  Tactical 
Decision  Aid  forecast  accuracy  and  mission  impacts  (e.g.,  TAWS  WOF  accuracy, 
weather  impacts  on  weapon  sensors);  and  (6)  forecast  performance  and  mission 
impacts  with  respect  to  specific  weather  factors  (e.g.,  surface  and  aloft  winds, 
dust,  fog).  Numerous  analyses  of  the  data  collected  indicate  that  weather 
forecast  provided  to  NSAWC  customers  have  significant  positive  impacts  on 
mission  planning  and  execution,  and  the  potential  to  have  additional  positive 
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impacts.  The  system  developed  in  this  study  can  be  readily  adapted  for  use  at 
other  operational  meteorology  and  oceanography  centers,  such  as  other  Naval 
METOC  and  Air  Force  Weather  units. 
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I.  INTRODUCTION 


A.  OVERVIEW 

The  Chief  of  Naval  Operations  (CNO)  stated  in  his  Guidance  for  2004 
(Clark  2004)  that  the  Navy  must:  “improve  output  metrics  to  better  define  our 
requirements  and  resource  needs,  and  instill  a  culture  of  improved  productivity  in 
everything  we  do.” 

In  his  2005  status  report  on  the  Naval  Oceanography  Program,  the 
Oceanographer  of  the  Navy,  RADM  Tomaszeski  wrote  (Tomaszeski  2004): 

Our  leaders  don’t  want  to  hear  how  important  it  is  to  describe  the 
environment,  or  provide  accurate  information.  Rather,  they  want  to 
hear  how  our  ‘enabling  capabilities’  translate  into  speed,  access,  or 
persistence  -  how  our  skills  results  in  advantages  in  force  posture 
(having  the  right  assets  in  the  right  place  at  the  right  time  for 
optimal  effect),  fewer  ships  sunk,  more  enemy  killed,  fewer  Blue 
Force  casualties,  less  time  spent  in  harm’s  way,  more  accurate 
placement  of  munitions,  etc. 

Over  the  last  few  years,  it  has  become  evident  that  metrics  must  become 
a  pivotal  tool  in  the  U.  S.  Naval  Meteorology  and  Oceanography  (METOC) 
community’s  efforts  to  improve  support  for  warfighters,  and  to  assist  in  the 
management  of  decreasing  resources  for  Naval  activities.  For  the  METOC 
community,  this  means  that  a  METOC  metrics  program  needs  to  be  developed 
and  used  as  a  tool  for  measuring  and  improving  the  community’s  performance 
and  productivity.  To  achieve  these  goals,  a  metrics  program  must  be  able  not 
only  to  evaluate  METOC  products,  but  also  to  determine  the  operational  impacts 
of  those  products  on  the  METOC  community’s  warfighting  customers. 

Several  prior  studies  have  indicated  that  METOC  metrics  programs  have 
the  potential  to  improve  METOC  support  to  warfighters  and  warfighter 
performance,  for  example,  by  increasing  the  efficiency  and  effectiveness  of 
METOC  and  warfighter  operations,  improving  operational  readiness,  and 
increasing  safety  by  decreasing  mishaps  (Cantu  2001,  Martin  2003,  Hinz  2004, 
Jarry  2005).  The  Naval  METOC  community  can  achieve  these  advantages  by 
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developing  a  metrics  program  that  accurately  gauges  the  impacts  of  METOC 
products  on  the  planning  and  conduct  of  warfighting  operations. 

Previous  METOC  and  Air  Force  Weather  (AFW)  metrics  studies  have 
been  completed  by  LCDR  Jake  Hinz,  USN,  and  Captain  Jeff  Jarry,  USAF,  while 
students  at  the  Naval  Postgraduate  School  (NPS).  LCDR  Hinz  developed  a  set 
of  statistical  tools  known  as  the  NPS  Metric  Method,  based  in  part  on  the 
National  Weather  Service  (NWS)  metrics  program.  Hinz  (2004)  applied  his  set  of 
tools  to  analyze  weather  forecasts  produced  during  Operation  Iraqi  Freedom 
(OIF)  and  to  estimate  the  impacts  of  those  forecasts  on  warfighting  operations. 
His  goal  was  to  provide  the  Commander  Naval  Meteorology  and  Oceanography 
Command  (CNMOC)  with  an  analysis  of  METOC  performance  and  impacts 
during  OIF,  and  a  general  method  for  evaluating  METOC  forecasts  and  their 
contributions  to  customer  operations. 

The  NPS  Metric  Method  has  proven  to  be  extremely  valuable  but  difficult 
for  most  operational  METOC  units  to  implement  in  their  daily  operations.  These 
difficulties  involved  problems  with  both  data  collection  and  data  analysis.  These 
problems  and  their  solutions  were  anticipated  by  Hinz  (2004),  who  noted: 

“Typically,  operational  requirements  and  the  high  tempo  of  forecast 
production  preclude  the  collection  of  METOC  forecast  data  and, 
especially,  observational  data  for  forecast  verification.  When  this 
data  is  collected,  it  is  rarely  tied  in  any  way  to  specific  missions.  It  is 
rarer  still  to  find  METOC  community  members  that  collect  data  on 
the  possible  impacts  of  METOC  conditions  and  forecasts  on 
operational  missions....  In  future  collection  efforts,  the  forecast, 
observation,  and  operations  data  collected  must  be  more 
standardized  in  their  formats,  and  have  a  clearly  defined  collection 
methodology.  This  will  allow  easier  application  of  metrics  methods, 
such  as  the  NPS  Metrics  Method,  and  faster  production  of  analyzed 
results  for  use  in  decision  making.” 

Jarry  (2005)  analyzed  the  performance  and  operational  impacts  of  mission 
execution  forecasts  (MEFs)  provided  by  USAF  Combat  Weather  Teams  (CWTs) 
to  the  Air  Mobility  Command  (AMC).  His  primary  goal  was  to  create 
standardized  procedures  and  analysis  techniques  to  ensure  that  AFW  metrics 
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are  relevant  to  operational  improvements  and  inline  with  customers’  operational 
requirements.  The  objectives  of  the  Jarry  (2005)  study  were  to: 

1 .  quantitatively  assess  the  performance  (e.g.,  accuracy,  skill)  of  AMC 
MEFs 

2.  determine  the  value  added  by  AMC  MEFs  to  warfighter  operations 

These  objectives  were  met  for  AMC  MEF  available  for  the  study  (MEFs 
from  fiscal  year  2003).  However,  the  data  collection  by  the  CWTs,  and  the 
analyses  by  Jarry  (2005)  were  very  labor  intensive,  and  the  results  of  the 
analyses  were  available  only  many  months  after  the  data  was  collected. 

The  results  of  both  the  Hinz  (2004)  and  Jarry  (2005)  studies  clearly 
indicate  there  is  a  need  for  a  more  automated  and  near-real  time  system  for 
collecting  and  analyzing  the  data  necessary  to  determine  forecast  performance 
and  operational  impacts.  The  purpose  of  this  study  was  to  develop  a  prototype 
of  such  a  system.  To  achieve  this  objective,  the  methods  developed  by  Hinz 
(2004)  and  Jarry  (2005)  were  merged  with  information  technology  tools  to 
develop  a  METOC  metrics  system  that  can  be  used  by  operational  METOC  and 
AFW  units  to: 

1 .  efficiently  collect  forecast,  observational,  and  operational  customer 
data 

2.  rapidly  analyze  that  data  to  produce  analyses  of  forecast  performance, 
customer  performance,  and  the  operational  impacts  of  the  forecasts 

3.  use  the  analyses  to  rapidly  produce  customized  metrics  reports  for 
METOC  units  and  their  customers 

B.  METOC  AND  AFW  METRICS 

METOC  and  weather  metrics  programs  have  been  developed  by  the 
NWS,  Naval  METOC  community,  and  AFW.  The  goals  and  basic  methods  of 
these  programs  are  similar,  but  there  are  significant  differences  in 
implementation.  The  status  of  the  NWS  program  has  been  reviewed  by  Hinz 
(2004)  and  Jarry  (2005).  This  section  summarizes  the  most  recent  efforts  in 
AFW  and  METOC  metrics. 
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1.  Navy  METOC  Metrics 

Hinz  (2004)  developed  a  set  of  metrics  tools  that  he  called  the  NPS  Metric 
Method.  He  developed  his  tools  based  on  previous  work  performed  by  the  NWS 
with  the  objective  of  providing  a  statistical  method  to  evaluate  weather  forecasts 
and  their  operational  impacts.  Hinz  used  Operation  Iraqi  Freedom  (OIF)  data 
compiled  during  March-April  2003.  His  objectives  were  to: 

1 .  develop  and  evaluate  a  set  of  metrics  for  evaluating  METOC  forecast 
performance 

2.  begin  the  exploration  of  METOC  impacts  on  OIF  air  operations 

3.  design  and  test  a  process  by  which  METOC  performance  and 
operational  impacts  can  be  measured  in  the  long  term  to  identify 
needed  improvements  and  challenges  to  be  overcome 

The  NPS  Metric  Method  is  based  on  a  contingency  table  that  contains 
data  on  forecasted  and  observed  conditions.  The  forecasted  and  observed 
conditions  are  categorized  as  either  red,  yellow,  or  green  based  on  a  known  set 
of  weather  criteria  for  a  given  type  of  mission.  The  comparisons  between 
forecast  and  observed  conditions  are  represented  by  forecast-observation  (FO) 
pairs.  There  are  nine  possible  pairs,  including,  for  example,  GG  representing  a 
forecast  condition  of  green  and  an  observed  condition  of  green;  GY  representing 
a  forecast  condition  of  green  and  observed  condition  of  yellow;  etc.  The  data  in 
the  table  represents  the  numbers  of  forecasts,  with  these  numbers  arranged 
according  to  the  corresponding  observations.  For  example,  12  red  forecasts  with 
corresponding  red  observations  (i.e. ,  12  RR  pairs);  15  red  forecasts  with 
corresponding  yellow  observations  (i.e.,  15  RY  pairs);  etc.  The  data  in  the 
contingency  table  can  be  analyzed  in  many  ways,  including  for  example: 

a.  Forecast  Accuracy  (FAC),  which  compares  the  number  of  correct 
forecasts  of  a  given  category  to  the  total  number  of  forecasts  in  that 
category  that  were  issued.  As  an  example,  FAC  for  red  forecasts 
would  be  calculated  by: 

FAC=100*  RR/(RR+RY+RG) 
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Here,  RR  represents  the  number  of  correct  red  forecasts  and  the 
denominator  represents  the  total  number  of  red  forecasts. 


b.  Probability  of  Detection  (POD),  which  compares  the  number  of 
correct  forecasts  of  a  given  type  of  event  to  the  number  of  events 
that  was  observed.  As  an  example,  the  POD  for  red  events  would 
be  calculated  by: 

POD=1 00*RR/(RR+YR+GR) 

Here,  the  denominator  represents  the  total  number  of  red  events. 

Hinz  (2004)  demonstrated  that  forecasts  for  combat  operations  could  be 
analyzed  to  determine  forecast  performance.  His  results  also  indicated  that  the 
operational  impacts  of  those  forecasts  could  be  quantitatively  assessed,  if 
additional  operational  performance  data  was  collected.  Hinz’s  methods,  results, 
and  recommendations  represent  a  major  advancement  beyond  what  had  been 
achieved  in  the  relatively  few  prior  METOC  metrics  efforts. 

2.  Air  Force  Weather  Metrics 

Currently,  the  AFW  metrics  program  is  managed  in  accordance  with  Air 
Force  Instruction  (AFI)  15-114,  Functional  Resource  and  Weather  Technical 
Performance  Evaluation  2001  (hereafter  referred  to  as  AFI  15-114(2001)). 
According  to  this  instruction,  the  goal  of  the  program  is  to  evaluate  the  overall 
health  of  the  AFW  system  and  to  understand  the  impacts  and  effectiveness  of 
the  weather  support  provided  at  every  level  of  military  operations.  This 
instruction  requires  that  three  types  of  forecast  verification  be  performed  by 
participating  units: 

1 .  forecast  impact  on  mission  execution  through  operational  verification 
(OPVER); 

2.  forecast  accuracy  by  terminal  aerodrome  forecast  verification 
(TAFVER); 

3.  resource  protection  effectiveness  through  warning/advisory  verification 
(WARNVER). 
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The  OPVER  or  mission  execution  forecast  verification  (MEFVER)  is 
performed  by  categorizing  the  forecast  according  to  a  set  of  Go  and  No  Go 
parameters,  with  Go  indicating  that  the  forecasted  weather  conditions  will  not 
negatively  impact  the  mission,  and  No  Go  indicating  that  the  mission  will  be 
impacted.  The  forecasted  weather  conditions  are  then  compared  to  what  was 
observed.  According  to  Jarry  (2005)  the  forecasting  units 

•  identify  the  weather  phenomena  that  are  most  critical  to  their 
operational  customers 

•  identify  the  No  Go  and  Go  thresholds  their  customers  want  to  have 
applied  to  those  phenomena 

•  provide  MEFs  for  those  phenomena 

•  provide  data  to  verify  those  MEFs 


In  2005,  Captain  Jeff  Jarry,  USAF,  performed  analysis  on  MEFVER  data 
provided  by  the  Air  Mobility  Command  (AMC).  The  statistical  analysis  consisted 
of  1 1  forecast  performance  and  four  operational  impact  metrics.  Jarry  (2005) 
defined  operator  performance  as: 

A  measure  of  the  success  or  quality  of  an  operation  based  on 
subjective  or  objective  methods  (e.g.,  number  of  missions 
cancelled,  number  of  missions  rescheduled,  number  of  missions 
flown  without  delays,  number  of  missions  delayed,  bomb  damage 
assessment,  amount  of  enemy  equipment  destroyed,  number  of 
training  hours  flown). 

and  operational  impact  metric  as: 

A  metric  for  determining  operational  impacts  (e.g.,  mitigation  rate, 
weather  delay  rate,  number  of  missions  saved  due  to  accurate 
forecasts,  etc.).  These  metrics  are  generally  determined  by 
comparing  operational  performance  data  to  weather  phenomena 
data  or  weather  forecast  performance  data. 

Additionally,  Jarry  (2005)  defined  the  following  impact  metrics. 

Mission  Weather  Delay  Rates:  The  weather  delay  rate  is  the  total 
number  of  missions  delayed  due  to  weather  divided  by  the  total 
number  of ...  missions. 
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Missions  at  risk  due  to  weather:  An  operational  impacts  metric  that 
describes  the  number  of  missions  that  forecasters  deemed  to  be  at 
risk  due  to  forecasts  of  adverse  weather  conditions  (e.g.,  due  to  a 
NoGoMEF). 

The  above  impact  metrics  were  applied  to  the  AMC  MEFVER  data  with 
the  goals  of  analyzing  the  accuracy  of  the  forecasts  provided,  and  the  impacts  of 
the  forecasts  on  warfighting  operations.  Jarry  (2005)  determined  that  “the  main 
return  on  investment  in  the  CWTs  probably  comes  from  their  forecasts  of 
relatively  uncommon,  but  mission  critical,  No  Go  conditions.”  Based  on  these 
results,  Jarry  (2005)  applied  several  more  operational  impacts  metrics  that 
provided  more  specific  analyses  of  No  Go  forecasts. 

At  one  of  the  AMC  units,  Global  Weather  Mobility  (WXM),  when  a  No  Go 
MEF  was  given  to  mission  planners,  the  forecasters  also  provided,  if  possible, 
with  a  recommendation  for  mitigating  the  negative  weather  impacts.  This 
recommendation  included  an  alternative  mission  plan  for  which  conditions  were 
forecasted  to  be  Go.  Data  was  collected  on  the  number  of  migration  forecasts 
that  were  provided  and  accepted  by  mission  planners.  These  results  were  then 
analyzed  to  determine  the  mitigation  rate  and  the  missions  saved  by  use  of  the 
mitigation  recommendations,  according  to  the  following  definitions: 

Mitigation  rate:  An  operational  impact  metric  that  describes  the 
frequency  at  which  mission  planners  accepted  the  advice  of 
forecasters  on  how  to  avoid  mission  delays  due  to  weather.  The 
mitigation  rate  is  the  number  of  mitigation  recommendations 
accepted  by  mission  planners  divided  by  the  number  of  mitigation 
recommendations  made,  all  times  100.  The  mitigation  rate 
describes  the  frequency  with  which  mitigation  recommendations 
were  accepted  by  mission  planners. 

Missions  saved:  An  operational  impact  metric  that  describes  the  net 
result  of  mitigation  recommendations  that  were  accepted  by 
mission  planners.  This  metric  accounts  for  the  number  of  mitigation 
recommendations  accepted  by  mission  planners,  the  accuracy  of 
the  MEFs  for  the  original  mission  plans,  and  the  accuracy  of  the 
MEFs  for  alternate  mitigation  plans. 
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From  these  results,  Jarry  (2005)  determined  that  additional  operational 
impacts  data  collection  was  needed  to  allow  for  a  thorough  metrics  program.  He 
recommended  that  data  on  the  following  issues  be  collected  to  allow  a  more 
complete  assessment  of  operational  impacts: 

•  missions  at  risk  due  to  weather 

•  missions  delayed  due  to  weather 

•  phenomena  that  placed  missions  at  risk 

•  phenomena  that  caused  mission  delays 

•  mitigations,  including  MEFs  for  original  and  alternate  plans 

•  planner  acceptance  and  rejection  of  mitigation  recommendations 

•  planner  reasons  for  accepting  and  rejecting  recommendations. 

C.  BASIC  PROBLEMS 

The  Hinz  (2004)  and  Jarry  (2005)  studies  highlighted  the  shortcomings  of 
current  techniques  used  to  collect  and  analyze  metrics  data,  and  made  several 
recommendations  to  improve  the  process.  Overall,  the  recommendations  can  be 
divided  into  four  main  categories: 

1 .  standardize  the  data  collected 

2.  automate  data  collection  and  analyses 

3.  reduce  the  complexity  of  the  system  for  participating  METOC  and  AFW 
units 

4.  reduce  the  time  needed  for  data  analysis  and  the  delivery  of  metrics 
results 

All  four  of  these  categories  are  inherently  linked.  For  example,  to 
decrease  the  time  required  to  process  the  data  to  achieve  near  real-time  results, 
automation  of  the  metric  system  is  required.  Likewise,  to  completely  automate  a 
metrics  system,  it  is  necessary  to  standardize  the  data  collection  process.  Thus, 
to  overcome  any  of  these  problems,  a  metric  program  must  make  progress  in  all 
four  categories. 
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1.  Data  Standardization 

Hinz  (2004)  found  that  data  standardization  is  a  basic  requirement  of  an 
effective  metric  program: 

This  standardization  will  allow  corporate  knowledge  and  brand 
recognition  to  be  portable  across  the  Navy.  Second,  data  from 
standardized  products  can  be  more  easily  inserted  into  databases 
for  analysis  and  archiving.  Third,  standardized  products  facilitate 
the  development  of  performance  benchmarks  that  span  the 
METOC  community  and  that  are  necessary  for  setting  overall 
goals,  identifying  needs  (e.g.,  in  research  and  development 
education  and  training),  and  creating  a  competitive  approach. 

Jarry  (2005)  also  noted: 

future  analyses  of  forecast  performance  and  operational  impacts 
data  would  greatly  benefit  from  more  consistent  and  complete  data 
from  the  different  forecasting  units. 

It  has  become  clear  to  implement  any  successful  metric  program  that  the 
data  to  be  collected  and  analyzed  must  be  standardized.  Without  consistent 
data  it  becomes  difficult  to  compare  data  from  different  forecasting  units  and  to 
develop  effective  benchmarks  of  performance.  Additionally,  using  uniform  data 
allows  for  more  effective  automation  of  data  collection  and  analysis  processes. 

2.  Automation 

Not  only  would  the  automation  of  a  metric  program  reduce  the  amount  of 
manpower  required  to  collect  and  process  the  data,  but  it  would  also  decrease 
the  amount  of  effort  require  by  the  managers  to  oversee  the  program.  Jarry 
(2005)  stated  that  a  fully  automated  metric  system  would  allow  managers  to 
“spend  more  time  making  management  decisions  and  conducting  training, 
instead  of  manually  recording  and  verifying  ...  data”. 

AFI  115-114  (2001)  calls  for  a: 

[metric]  system  that  automates  all  metrics,  from  data  collection  and 
aggregation  to  data  quality  control.  The  end-state  will  be  an 
automated  web-based  system  with  the  capability  to  provide  ad  hoc 
analyses  and  reports  (assessments)  for  all  levels  of  AFW  support. 
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Funding  and  technology  shortfalls  and  the  lack  of  operational 
effectiveness  databases  require  a  phased  approach  in  order  to 
achieve  this  vision. 

3.  Complexity 

In  the  summer  of  2004,  LCDR  Marc  Steiner,  the  USS  Saipan  METOC 
officer,  contacted  Prof.  Tom  Murphree  at  NPS  about  developing  a  METOC 
metrics  program  for  the  Saipan’s  upcoming  fall  2004  deployment.  We  (the 
author,  Prof.  Murphree,  and  NPS  staff)  developed  a  program  for  the  Saipan 
using  the  NPS  Metrics  Method  developed  by  Hinz  (2004).  Data  was  collected 
and  analyzed  and  reports  were  issued  from  NPS  to  the  Saipan.  However,  the 
results  were  incomplete,  largely  due  to  the  complexity  of  the  process,  for  both: 
(a)  Saipan  personnel  collecting  the  forecast  and  observational  data,  and  using 
the  resulting  metrics;  and  (b)  NPS  personnel  processing  and  analyzing  the  data, 
and  issuing  metrics  reports.  In  particular,  the  shipboard  staff  had  problems 
applying  the  relatively  academic  concepts  employed  in  the  NPS  Metric  Method 
tool  set.  Additionally,  the  USS  Saipan  group  did  not  have  the  extra  time  it 
needed  to  completely  fully  engage  in  such  a  project.  The  end  result  was  that  the 
Saipan’s  METOC  division  was  only  able  to  collect  a  relatively  limited  amount  of 
data.  At  NPS,  the  analysis  of  the  data  and  delivery  of  reports  to  the  Saipan  was 
inhibited  by  the  time  required  to  ingest  and  analyze  the  data  and  issue  metrics 
results.  These  problems  were  due  to  the  awkwardness  of  the  format  in  which  the 
data  was  collected  and  delivered  (via  emails  and  postal  mail  from  the  Saipan) 
and  analyzed  (via  Excel  spreadsheets  and  graphics).  The  NPS  side  of  the 
process  was  an  adaptation  of  the  system  used  by  Hinz  (2004).  This  system  was 
effective  for  an  academic  research  approach  but  proved  to  be  difficult  to  adapt  to 
an  operational  setting.  Our  conclusion  was  that  a  new  system  was  needed  that 
would  reduce  the  level  of  information  technology  (IT)  and  data  analysis 
sophistication  needed  by  participating  units  (e.g.,  the  Saipan),  and  reduce  the 
amount  of  effort  needed  by  NPS  personnel.  This  new  system  should  be  based 
on  the  basic  concepts  used  by  Hinz  (2004)  and  Jarry  (2005)  for  assessing 
forecast  performance  and  the  operational  impacts  of  forecasts. 
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4.  Time  Required 

AFI  15-114  (2001)  dictates  several  phases  for  achieving  the  end  goal  of  a 
fully  automated  metrics  program.  Phase  3  of  the  plan  calls  for  the  automation  of 
the  operational  verification  (OPVER)  program  that  would  include  web-based  ad 
hoc  analysis  and  reports  that  allow  the  operational  customer  access  to  near  real 
time  results. 

In  the  Hinz  (2004)  and  Jarry  (2005)  studies,  the  time  from  the  final 
collection  of  the  metrics  data  from  operational  units  until  a  final  metrics  report 
was  completed  ranged  from  six  to  sixteen  months.  In  part,  these  large  lag  times 
were  due  to  the  academic  context  in  which  the  studies  were  conducted.  In  this 
context,  the  development  and  application  of  concepts  was  emphasized,  and  the 
goal  was  to  lay  the  foundation  for  an  operational  system  that  would  be  developed 
and  implemented  in  subsequent  studies.  The  Hinz  and  Jarry  studies  were 
successful  in  reaching  this  goal,  but  our  initial  work  with  the  Saipan  made  it  clear 
that  for  metrics  program  to  be  effective,  the  turn  around  time,  from  collection  of 
data  to  delivery  of  metrics  results,  must  be  drastically  decreased,  with  the  end 
goal  being  near  real-time  results. 

D.  SPECIFIC  QUESTIONS  OF  THIS  STUDY 

1.  What  Data  Needs  to  be  Collected  to  Determine  Operational 
Impacts? 

One  of  the  major  steps  in  developing  a  metrics  program  is  determining 
what  data  must  be  collected  to  effectively  determine  operational  impacts.  AFI 
15-114  (2001 )  dictates  that  the  weather  support  team  and  supported  commander 
will  “  operationally  define  weather  support  effectiveness  measures  (metrics)  and 
mechanisms  to  cross  feed  these  metrics  to  weather  support  providers  in  a  timely 
manner.  “ 

Additionally,  AFI  15-114  (2001)  provides  the  following  suggestions  on 
what  mission  and  weather  data  could  be  collected  to  determine  operational 
impacts: 

1 .  Total  number  of  scheduled  missions 

2.  Total  number  of  missions  cancelled  due  to  correct  forecasts 
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3.  Total  number  of  missions  cancelled  despite  correct  forecasts 

4.  Total  number  of  missions  re-targeted,  rearmed  (different  weapons),  or 
rescheduled  due  to  correct  forecasts 

5.  Total  number  of  missions  re-targeted,  rearmed  (different  weapons),  or 
rescheduled  despite  correct  forecasts 

6.  Total  number  of  missions  non-effective  or  partially  effective  due  to 
actual  weather  conditions 

7.  Total  number  of  missions  non-effective  or  partially  effective  due  to 
incorrect  forecasts 

8.  For  missions  that  were  non-effective  or  partially  effective  due  to  actual 
weather  conditions  or  incorrect  forecasts,  identify  the  specific  parts  of 
the  missions  that  were  affected 

In  this  study,  we  addressed  the  methods  for  collecting  and  analyzing  all  of 
these  types  of  data,  plus  additional  data  for  assessing  forecast  performance,  the 
operational  impacts  of  forecasts,  and  the  operational  impacts  of  specific  weather 
phenomena. 

2.  How  Can  Technology  be  Used  to  Increase  Automation  and 
Turn  Around  Times? 

Leveraging  information  technology  (IT)  is  an  obvious  answer  to  solving  the 
problems  of  lack  of  automation  and  standardization,  the  complexity  of  metric 
programs,  and  the  lag  time  for  final  reports.  But  it  is  not  obvious  what  IT  needs 
to  be  leveraged,  or  how  that  leveraging  should  be  done  to  be  both  efficient  and 
effective.  What  is  known  is  that  a  computer  based  system  is  needed  that  would 
allow  for:  (a)  the  collection  of  metric  data  with  minimal  customer  effort;  (b)  the 
near-automatic  processing  of  the  data;  and  (c)  the  rapid  dissemination  of  results 
back  to  METOC  and  AFW  units  and  their  customers.  In  this  study,  we  devoted  a 
large  amount  of  effort  to  identifying,  testing,  and  implementing  IT  tools  for 
achieving  these  goals. 

3.  What  Data  Collection  and  Analysis  Methods  Provide  the 
Clearest  Quantitative  Assessments  of  Operational  Impacts? 

One  of  the  basic  problems  of  any  metrics  program  is  determining:  (a)  how 

to  collect  the  most  useful  data,  especially  operational  impacts  data;  (b)  which 
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analytical  methods  work  best  with  the  data  collected;  and  (c)  which  analytical 
methods  work  best  for  the  participating  METOC  and  AFW  units  and  their 
operational  customers.  With  the  wide  range  of  support  products  provided  by 
military  weather  units,  these  issues  can  be  very  difficult  to  resolve.  The  data 
collection  and  analysis  methods  used  by  Hinz  (2004)  and  Jarry  (2005)  worked 
well  for  their  particular  military  operations,  units,  and  customers.  But  these 
methods  are  probably  not  appropriate  for  many  types  of  operational  impacts 
data,  and  for  many  METOC  and  AFW  units,  that  were  not  considered  in  the  Hinz 
and  Jarry  studies.  In  this  study,  we  developed  several  methods  for  data 
collection  and  analysis,  with  an  emphasis  on  developing  a  system  that  is  robust 
and  readily  adapted  to  different  units  and  their  customers. 

E.  GOALS  OF  THIS  STUDY 

This  study  had  four  main  goals,  all  focused  on  overcoming  the  basic 
problems  outlined  in  the  preceding  sections.  The  first  goal  was  to  identify  the 
data  necessary  for  quantitatively  determining  forecast  performance  and  the 
operational  impacts  of  those  forecasts.  We  pursued  this  goal  by:  (a)  leveraging 
what  had  been  learned  about  data  requirements  in  the  Hinz  (2004)  and  Jarry 
(2005)  studies;  and  (b)  working  with  the  intended  users  of  the  metrics  system 
developed  in  this  study  (i.e.,  METOC  and  AFW  units)  to  determine  what  types  of 
warfighter  support  they  provided,  what  products  they  issued,  what  data  they  were 
able  to  collect,  and  what  metrics  results  they  and  their  customers  needed. 

The  second  goal  was  to  develop  an  online  data  collection  system  linked  to 
an  online  database  for  archiving  and  analyzing  the  required  data  into  a  readily 
accessible  form.  Such  a  system  would  be  well  suited  for  collecting  and  analyzing 
standardized  data.  Additionally,  many  of  the  data  quality  problems  that  Hinz 
(2005)  found  could  be  eliminated  by  validating  the  data  as  it  is  being  submitted. 
Hinz  (2005)  stated  that: 

There  were  some  significant  data  quality  challenges  in  this  study. 

These  included  problems  with  incompatible  formatting,  the  need  to 
develop  databases  prior  to  conducting  data  analyses,  and 
uncertainties  about  the  quantity  and  quality  of  the  observations 
used  for  verification. 
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The  third  goal  was  to  develop  a  web-based  interface  that  would  provide 
users  with  near  real-time  access  to  the  metrics  results  (i.e.,  the  results  of  the  data 
analyses).  One  of  the  basic  problems  with  previous  metrics  efforts  at  NPS  was 
the  delay  between  the  data  collecting  and  the  final  metrics  reports.  A  web  based 
system  for  collecting  and  analyzing  data,  and  reporting  metrics  results,  would 
allow  users  access  to  their  metrics  results  immediately  after  their  data  has  been 
entered. 

The  fourth  goal  was  to  develop  a  final  metrics  report  format  that  fully 
supported  the  needs  of  the  users  (i.e.,  the  METOC  and  AFW  units  and  their 
operational  customers).  This  goal  was  developed  based  on  the  concept  that 
users  know  better  than  anyone  else  what  they  need.  We  would  provide  guidance 
to  users  on  what  is  technically  and  statistically  feasible,  and  work  with  users  to 
develop  an  end  product  that  is  accurate,  complete,  and  useful. 
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II.  DATA  AND  METHODS 


A.  USS  SAIPAN 

In  the  summer  of  2004,  the  METOC  division  onboard  the  USS  Saipan 
contacted  NPS  and  asked  for  assistance  in  the  development  of  system  to 
measure  the  performance  of  their  locally  generated  forecasts,  in  particular,  the 
accuracy  of  its  forecasts.  In  response,  we  proposed  to  the  USS  Saipan  to 
develop  a  web  based  system  to  provide:  (a)  archiving  and  analysis  of  the 
forecasted  and  observed  weather  conditions;  and  (b)  delivery  of  the  results  from 
the  data  analyses.  The  data  collection  and  analyses  were  designed  so  that  the 
metrics  methods  used  by  Hinz  (2004)  could  be  readily  adapted.  Our  goal  was  to 
develop  and  test  a  system  for  implementing  operationally  the  methods  used  by 
Hinz.  Our  focus  was  on  creating  a  system  that  would  work  for  operational  units 
such  as  the  Saipan,  not  on  conducting  a  thorough  analysis  of  the  Saipan 
forecasts. 

1.  Data 

The  data  consisted  of  a  daily  forecast  produced  onboard  the  USS  Saipan 
and  corresponding  hourly  weather  observations  from  September  and  October 
2004.  The  daily  forecast  was  a  stoplight  (red,  yellow,  green)  forecast  valid  each 
day  at  1200  UTC  with  lead  times  from  6  to  96  hours  (Figure  1).  The  forecast  and 
observation  data  were  available  for  five  types  of  operations: 

•  Aviation  Operations 

•  Landing  Craft  Air  Cushion  (LCAC)  Operations 

•  Landing  Craft  Utility  (LCU)  Operations 

•  Replenishment  At  Sea  (RAS)  Operations 

•  Smallboat  Operations 

For  each  operation  type,  the  forecasts  for  a  given  valid  time  were  grouped 
together,  with  red,  yellow,  or  green  forecasts  for  each  6  hour  period  from  6  to  96 
hours  prior  to  the  valid  time. 

Weather  observations  were  also  collected  on  an  hourly  basis.  The 
observations  include  the  following  weather  parameters: 
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•  Time  (UTC) 

•  Wind  speed  (knots) 

•  Visibility  (nautical  miles) 

•  Weather  phenomena 

•  Sky  condition  to  include  cloud  density  and  height 

•  Combined  seas  (feet) 

The  complete  data  set  consisted  of  all  the  forecasts  provided  while  the 
USS  Saipan  was  underway  and  the  corresponding  hourly  observations  for  the 
months  of  September  and  October  2004.  The  total  number  of  forecasts  used  for 
this  study  was  1440  and  observations  were  569. 

2.  Methods 

a.  Flow  of  Data  and  Analysis  Results 

A  web  interface  was  designed  that  allowed  the  USS  Saipan  to 
remotely  enter  both  forecast  and  observation  data  into  a  database  located  at 
NPS  (see  http://wx.met.nps.navy.mil/~mdbutler/index.html  and  Figures  2  and  3, 
please  note  that  areas  of  the  website  are  password  protected).  The  data  was 
then  inserted  into  a  database  using  a  scripting  code  which  was  embedded  in  the 
web  page.  The  web  interface  allows  users  to  review  data  that  had  been 
previously  entered  and  to  delete  any  data  that  may  have  been  entered 
incorrectly.  To  accomplish  this,  the  following  computer  languages  were  used: 

•  Hypertext  Markup  Language  (HTML) 

•  PHP:  Hypertext  Preprocessor  (PHP) 

•  My  Structured  Query  Language  (MySQL) 

To  better  understand  why  these  languages  were  chosen,  a  brief 
explanation  of  each  of  them  is  helpful.  All  three  are  open  source  languages  or 
languages  that  can  be  used  and  manipulated  for  free  as  long  as  it  is  not  for 
commercial  gain. 

HTML  is  the  basic  language  for  the  development  of  web  pages.  It 
was  chosen  over  other  more  powerful  options  such  as  Extensible  Hypertext 
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Markup  Language  (XHTML)  because  of  its  ease  of  use  and  the  simplicity  of  the 
web  pages  that  needed  to  be  designed. 

PHP  is  a  widely  used  server-side  scripting  language  that  can  be 
embedded  directly  into  a  HTML  webpage.  Server-side  scripting  refers  to 
computer  programs  that  are  located  in  the  webpage  but  are  actually  run  on  the 
web  server.  By  embedding  the  scripting  code  in  the  web  page,  it  allows  a 
developer  to  create  a  dynamic  web  page  that  is  capable  of  processing  user 
supply  information  and  interacting  directly  with  a  database. 

MySQL  is  the  most  popular  open  source  database  system.  MySQL 
is  designed  to  be  a  relational  database,  which  means  the  data  is  stored  in 
individual  tables  versus  being  stored  as  one  large  group.  This  storage  method 
allows  MySQL  to  process  data  faster  and  more  efficiently  than  otherwise. 

All  three  of  the  main  program  languages  used  had  the  following 
additional  advantages: 

1 .  Cost.  All  are  free  for  use  in  the  non-commercial  environment.  The 
cost  of  similar  commercial  software  could  easily  be  several  thousands 
of  dollars. 

2.  Simplicity.  All  three  languages  are  relatively  easy  to  learn  and  use. 

3.  Flexibility.  HTML  web  pages  are  the  standard  for  web  pages  and  can 
be  viewed  on  any  web  browser.  PHP  and  MySQL  can  be  operated  on 
UNIX,  MAC,  or  Windows  operating  systems.  PHP  is  also  compatible 
with  almost  any  major  web  server  software.  For  this  study,  PHP  was 
installed  on  Apache,  Microsoft  Internet  Information  Server  (IIS),  and 
Solaris. 

4.  Speed.  Application  speed  was  critical  for  the  web  system  developed 
for  this  project  since  the  system  was  to  be  accessed  by  units  underway 
with  poor  and  slow  connections.  One  of  the  advantages  of  using  this 
language  configuration  is  that  PHP  is  a  server-side  application.  This 
results  in  a  drastic  time  savings  for  the  types  of  programs  developed 
for  this  study. 
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b.  Analysis  and  Output 

Once  the  web  interface  was  completed,  the  data  was  entered  into 
the  database.  At  this  point,  a  PERL  executable  script  was  used  to  convert  the 
hourly  observations  to  red,  yellow,  and  green  format  to  allow  comparison  to  the 
forecasted  data.  The  forecast  information  and  converted  observations  were  then 
exported  to  an  Excel  file  for  comparison  and  analysis  to  be  conducted  in 
accordance  with  the  NPS  Metrics  Method  developed  by  Hinz  (2004).  The 
following  metrics  were  obtained  from  the  data: 

•  Forecast  Accuracy  (FAC) 

•  Probability  of  Detection  (POD) 

•  Number  of  Accurate  Forecasts  (NAF) 

FAC  and  POD  were  plotted  against  lead  time  for  each  forecast 
condition  and  for  each  operation  type.  For  example,  a  FAC  plot  for  aviation 
operations  consists  of  curves  representing  the  accuracy  of  the  red,  yellow,  and 
green  forecasts,  and  the  mean  of  these  forecasts  (the  total  FAC),  at  each  lead 
time.  Figures  17  and  18  present  FAC  plots  for  aviation  and  LCAC  forecasts, 
respectively,  and  Figures  22  and  23  depict  POD  for  the  same  forecast  types. 
NAF  was  defined  as  the  sum  of  accurate  forecasts  provided  over  a  given  time 
frame. 

B.  NAVAL  PACIFIC  METEOROLOGY  AND  OCEANOGRAPHY 

DETACHMENT,  FALLON,  NEVADA 

In  March  of  2005,  we  began  discussions  with  LCDR  Alex  Cantu,  METOC 
OIC  at  Naval  Pacific  Meteorology  and  Oceanography  Detachment,  Fallon, 
Nevada,  (hereafter  referred  to  as  NPMOD)  about  the  possibility  of  automating 
and  analyzing  the  data  that  NPMOD  was  collecting  to  determine  the  operational 
impacts  of  the  weather  support  that  it  was  providing  to  its  main  customer,  the 
Naval  Strike  and  Air  Warfare  Center  (NSAWC)  at  Fallon.  NPMOD  had 
developed  a  basic  form  for  collecting  information  about  the  operational  impacts 
that  resulted  from  the  weather  forecasts  and  recommendations  it  provided  for  the 
planning  of  NSAWC  missions. 
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The  role  of  NPMOD  in  the  planning  of  NSAWC  mission  is  shown  in 
Figures  4  and  5  (Cantu  2005).  Understanding  NPMOD’s  role  is  critical  in 
determining  the  types  and  amounts  of  METOC  and  customer  data  that  can  be 
collected,  and  the  types  of  analyses  that  can  be  conducted.  For  example, 
because  of  the  role  of  NPMOD  forecasters’  role  in  providing  forecasts  and 
recommendations  to  mission  planners  several  days  prior  to  mission  takeoff,  it  is 
possible  to  collect  data  that  allows  us  to  determine  the  number  of  missions  saved 
and  other  operational  impacts.  Thus,  we  recommended  to  NPMOD  personnel 
that  they  adapt  the  mission  saved  concept  described  by  Jarry  (2005)  to 
determine  how  their  forecasts  and  recommendations  impacted  NSAWC  mission 
planning  and  execution,  including  such  things  as  changes  in  mission  schedule, 
weapons  selection,  target  selection,  and  tactics  selection. 

Our  discussions  led  to  the  development  of  a  plan  for  us  to  collaborate  with 
NPMOD  on  developing  a  system  for  collecting  data,  objectively  analyzing  the 
data,  and  producing  metrics  reports  in  near  real  time.  This  collaboration  is  still 
underway  as  of  the  writing  of  this  report.  The  key  objectives  of  this  collaboration 
are  assessments  of  the: 

1 .  performance  of  the  NPMOD  local  air  field  forecast  for  Naval  Air  Station 
(NAS)  Fallon 

2.  weather  phenomena  that  affect  NSAWC  missions  being  flown  at  NAS 
Fallon  ranges 

3.  operational  impacts  of  forecast  provided  by  NPMOD  to  NSAWC  during 
planning  and  mission  execution 

Our  goal  in  working  with  NPMOD  was  and  is  to  develop  and  test  a  data 
collection  and  analysis  system  that  builds  on  the  lessons  learned  from  the  Hinz 
(2004)  and  Jarry  (2005)  studies,  and  from  our  work  with  the  Saipan.  As  in  the 
Saipan  work,  our  focus  for  this  study  was  on  creating  a  system  that  would  work 
for  operational  units  such  as  the  Saipan,  not  on  conducting  a  thorough  analysis 
of  the  NPMOD  forecasts.  We  expect  that  these  more  comprehensive  analyses 
will  be  done  in  future  NPS  studies,  especially  once  larger  quantities  of  data  have 
been  collected  by  the  system. 
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1 .  Data 

a.  Original  NPMOD  Data  Collection 

Prior  to  the  development  of  the  NPS-NPMOD  collaboration, 
NPMOD  was  collecting  information  based  on  the  following  topics  and  answers  to 
the  following  questions: 

•  Debrief  time  and  location 

•  Mission  type 

•  Did  weather  impact  the  mission? 

•  Was  the  mission  changed  due  to  weather? 

•  Were  the  weapons  load-out  changed  due  to  weather? 

•  How  did  weather  impact  the  mission  and  what  was  the  impact? 

NPMOD  was  developing  procedures  for  analyzing  this  information  when  our 
discussions  with  NPMOD  began  (Cantu  2005).  These  discussions  led  to  a  major 
revision  of  the  data  collection  process  and  the  development  of  a  data  analysis 
scheme. 

b.  Revised  NPMOD  Data  Collection 

Our  work  with  NPMOD  allowed  us  to  pursue  one  of  the  basic  goals 
of  this  study  was  to  identify  the  types  of  data  necessary  to  objectively  and 
quantitatively  determine  operational  impacts.  To  do  so,  we  applied  the  lessons 
learned  from  the  Hinz  (2004)  and  Jarry  (2005)  studies  to  the  context  in  which 
NPMOD  was  providing  METOC  support  to  NSWAC. 

One  of  these  lessons  was  that  operational  impacts  of  forecasts 
could  not  be  determined  without  knowing  the  performance  of  those  forecasts 
(e.g.,  their  FAC  and  POD).  This  led  us  to  recommend  to  NPMOD  personnel  that 
they  expand  their  data  collection  efforts  to  include  forecast  and  observational 
data,  so  that  forecast  performance  could  be  assessed  and  compared  to 
operational  performance.  Without  knowing  the  accuracy  of  the  forecasts 
provided,  it  is  impossible  to  fully  determine  how  the  forecasts  affected  mission 
planning  and  execution.  For  example,  if  negative  impacts  (red  conditions)  were 
forecasted  to  occur  during  a  mission,  and  if  the  mission  plan  was  changed  in 
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response  to  this  forecast,  then  it  would  be  critical  to  know  if  the  weather  forecast 
was  accurate  or  not.  If  the  forecast  was  accurate,  then  the  mission  could  be 
classified  as  a  mission  saved,  meaning  that  the  forecast  enabled  planners  to 
revise  the  mission  and  avoid  adverse  weather  conditions.  If  the  forecast  was 
incorrect,  then  mission  planners  were  incorrectly  guided  by  the  forecast,  and  the 
mission  plan  was  unnecessarily  altered. 

Our  discussions  with  NPMOD  led  to  extensive  changes  in  the  data 
collection  process;  in  particular,  the  collection  of  data  based  on  the  following 
topics  and  answers  to  the  following  questions: 

•  Air  group  flying 

•  Mission  type 

•  Date  and  time  of  mission  takeoff 

•  During  the  initial  team  planning,  what  weather  was  forecasted  to 

negatively  impact  the  mission? 

•  What  changes  to  the  mission  plan  resulted  from  forecasted 
negative  impacts? 

•  Were  these  changes  in  the  mission  plan  correct  for  the  given  the 
forecast? 

•  Were  these  changes  in  the  mission  plan  given  the  actual  weather 
encountered? 

•  Were  these  changes  necessary? 

•  What  were  the  actual  negative  weather  impacts  on  the  mission? 

•  What  were  the  weather  phenomena  that  negatively  impacted  the 
mission? 

•  What  changes  to  the  weapons  plan  resulted  from  Tactical 
Acquisition  Weapons  Software  (TAWS)  predictions? 

•  Were  the  TAWS  predictions  accurate? 

The  data  collection  process  was  also  revised  to  collect  data  on 
forecasted  and  observed  sustained  wind  speed,  ceiling,  and  visibility  at  the 
takeoff  and  landing  air  field.  This  data  was  collected  for  use  in  analyzing  the 
performance  of  the  air  field  forecasts.  We  discussed  with  NPMOD  personnel  the 

collection  of  data  for  analyzing  the  performance  of  the  forecasts  of  enroute  and 
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target  weather  in  the  Fallon  ranges.  Unfortunately,  very  little  or  no  in  situ 
observational  data  is  currently  available  from  the  ranges  (aside  from  pilot  reports) 
(Cantu  2005).  So  we  were  not  able  to  include  this  critical  data  in  this  study. 
However,  the  process  developed  in  this  study  for  collecting  and  analyzing 
forecast  and  observational  data  for  the  air  field  will  be  directly  applicable  to  range 
data  when  it  becomes  available. 

The  data  collection  process  is  illustrated  in  Figures  6-14  (see  also 
NPS  METOC  Metrics  Support  Site,  http://wx.met.nps.navy.mil/ 
metrics/index. html).  These  figures  show  the  nine  main  data  categories  (Figure  6) 
and  the  associated  portions  of  the  online  data  collection  form  for  each  of  the  nine 
categories  (Figures  7-14).  For  example,  Figure  9  shows  the  data  collected 
regarding  mission  changes  made  during  the  team  planning  phase  of  the  NSAWC 
mission  planning  process  (see  Figures  4  and  5). 

The  data  set  collected  for  this  study  and  used  to  test  the  metrics 
data  collection  and  analysis  system  developed  for  NPMOD  consisted  of  data  for 
49  missions  conducted  by  two  carrier  air  wings  during  operations  at  NSAWC  in 
May  and  June  2005.  The  data  collection  and  analysis  system  is  still  being  used 
by  NPMOD  (Cantu  2005)  and  will  be  collecting  additional  NSACW  mission  data 
during  2005  and  2006.  This  data  will  be  analyzed  in  future  NPS  METOC  metrics 
studies. 

c.  Forecast  Verification 

The  planning  forecasts  issued  during  the  planning  cycle  were 
verified  mainly  by  using  pilot  reports  of  weather  conditions  during  their  missions. 
These  reports  were  obtained  by  NPMOD  personnel  during  mission  debriefs  (see 
Figures  4  -  5).  The  focus  of  these  verifications  was  on  verifying  the  categorical 
(red,  yellow,  green)  forecast  of  negative  weather  impacts.  The  air  field  forecasts 
were  verified  using  data  collected  by  NPMOD  observers  and  by  an  automated 
surface  observing  system  (ASOS)  at  the  NAS  Fallon  air  field.  The  focus  of  these 
verifications  was  on  verifying  sustained  wind  speed,  minimum  ceilings,  and 
visibility. 
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2.  Methods  -  Flow  of  Data  and  Information 

The  methods  used  in  collecting  and  archiving  the  data  for  this  portion  of 
the  study  were  similar  to  those  used  in  collecting  the  data  from  the  USS  Saipan. 
The  data  was  entered  into  a  HTML  web  page  form.  Then,  using  PHP  code 
embedded  into  the  web  page,  the  data  was  entered  into  a  MySQL  database. 

The  key  difference  between  the  Saipan  and  NPMOD  portions  of  this  study 
was  that  the  Saipan  data  was  processed  outside  of  the  web  based  system,  while 
the  NSAWC  data  was  processed  in  an  on-demand  format.  The  on-demand 
results  were  obtained  by  embedding  the  computer  programs  used  to  perform  the 
analyses  directly  into  the  results  or  output  web  page.  The  end  result  was  users 
can  view  the  data  analysis  results  immediately  after  entering  the  data. 

3.  Methods  -  Analyses  and  Output 

By  applying  the  techniques  developed  by  Hinz  (2004)  and  Jarry  (2005), 
several  forecast  performances  and  operational  impacts  metrics  and  other  tools 
were  included  in  the  analysis  part  of  the  web  based  system  for  analyzing  the  data 
collected.  The  analyses  were  divided  into  three  categories:  (a)  operational 
impacts;  (b)  mission  planning  forecast  performance  and  operational 
performance;  and  (c)  air  field  forecast  accuracy  and  probability  of  detection, 
a.  Operational  Impacts 

Three  main  metrics  were  developed  to  assess  the  operational 
impacts  of  the  forecasts  provided  during  mission  planning:  (a)  missions  saved; 
(b)  weapons  saved;  and  (c)  missions  and  weapons  potentially  saved.  These 
three  metrics  are  explained  below: 

Missions  saved.  A  mission  is  considered  to  have  been  saved  if:  (a) 
planners  changed  the  mission  in  response  to  an  accurate  weather 
forecast;  and  (b)  the  mission  would  have  been  negatively  impacted 
by  weather  had  the  mission  not  been  changed.  The  possible 
mission  changes  include  changes  in  mission  schedule,  target 
selection,  and  tactics  selection. 

Weapons  saved.  Weapons  are  considered  to  have  been  saved  if: 

(a)  planners  changed  the  weapons  in  response  to  an  accurate 
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weather  forecast;  and  (b)  the  weapons  would  have  been  negatively 
impacted  by  weather  had  the  weapons  not  been  changed. 

Missions  and  weapons  potentially  saved.  Missions  and  weapons 
are  considered  to  have  been  potentially  saved  if:  (a)  an  accurate 
forecast  of  negative  impacts  was  provided  to  mission  planners;  (b) 
the  planners  chose  not  to  alter  the  mission  or  weapons  in  response 
to  the  forecast;  and  (c)  the  mission  or  weapons  selected  were 
negatively  impacted  by  weather. 

Several  additional  metrics  were  developed  to  provide  more  detailed 
analyses  of  the  data  that  was  collected.  These  metrics  were  used  to  determine: 
(a)  the  weather  phenomena  forecasted  and  observed  to  cause  negative  impacts; 
and  (b)  the  negative  mission  impacts  that  resulted  from  these  weather 
phenomena. 

Weather  phenomena  forecasted  to  have  negative  impacts:  These 
are  phenomena  that  were  forecasted  to  impact  missions  and 
weapons,  including:  excessive  surface  winds;  excessive  winds 
aloft;  altitude  restrictions  due  to  turbulence,  icing,  or  thunderstorms; 
reduced  surface  visibility  due  to  fog,  haze,  dust  storm,  or 

precipitation;  or  low  thermal  contrast. 

Weather  phenomena  observed  to  have  negative  impacts:  These 
are  the  weather  phenomena  that  actually  impacted  missions  and 
weapons,  including:  excessive  surface  winds;  excessive  winds 
aloft;  altitude  restrictions  due  to  turbulence,  icing,  or  thunderstorms; 
reduced  surface  visibility  due  to  fog,  haze,  dust  storm,  or 

precipitation;  or  low  thermal  contrast. 

Initial  team  planning  changes:  These  are  the  changes  that  were 
made  to  the  missions  as  a  result  of  forecasts  provided  during  the 

planning  phase.  These  changes  include:  event  delayed  or 

rescheduled,  weapons  changed  for  high  winds,  weapons  changed 
for  visibility  or  ceiling,  or  other. 

Negative  impacts  resulting  from  weather:  These  are  the  impacts 
that  occurred  during  mission  execution  as  a  result  of  the  weather 
encountered  on  the  range.  These  impacts  were  divided  into  three 
categories:  (a)  changes  in  schedule;  (b)  changes  on  type  of  war 
(changes  in  type  of  exercise  conducted),  and  (c)  partial  mission 
(reduction  in  mission  scope). 

b.  Mission  Planning  Forecast  Performance  and  Operational 
Performance 


24 


We  compared  the  accuracy  of  the  mission  planning  forecasts 
provided  to  planners  several  days  before  mission  takeoff  (see  Figures  4  and  5) 
with  the  mission  planning  changes  to  determine  the  impacts  of  these  forecast  on 
planning.  The  planning  forecasts  and  planning  changes  were  organized  into  one 
of  the  following  five  categories: 

1 .  Weather  was  forecasted  correctly  and  the  forecast  led  to  a  useful 
mission  change 

2.  Weather  was  forecasted  correctly  but  mission  planners  made  no 
change 

3.  Weather  forecast  was  incorrect  and  led  to  an  unnecessary  mission 
change 

4.  Weather  forecast  was  incorrect  but  mission  planners  made  no  change, 
and  a  correct  forecast  would,  if  accepted  by  planners,  have  led  to  a 
necessary  mission  change 

5.  Not  enough  information  to  determine  if  the  mission  change  was  useful 
or  unnecessary 

Additionally,  data  for  the  above  five  categories  was  subdivided 
according  to  the  weather  phenomena  that  was  forecasted  or  observed  to  have  a 
negative  impact.  For  example,  missions  which  fell  into  category  1,  above,  were 
analyzed  to  determine  the  number  of  missions  for  which  accurate  forecasts  of 
excessive  surface  winds  were  issued  and  led  to  a  useful  mission  change. 

c.  Air  Field  Forecast  Accuracy  and  Probability  of  Detection 
FAC  and  POD  were  calculated  for  the  local  air  field  forecasts, 
including  FAC  and  POD  for  ceiling,  visibility,  and  winds. 

4.  Overview  of  Data  Collection  and  Analysis 

The  overall  process  for  collecting  and  analyzing  data,  and  for  issuing 
analysis  reports,  is  shown  in  Figure  4.  This  figure  may  be  deceptive,  since  the 
development  of  this  process  required  us  to  overcome  significant  conceptual 
challenges  in  determining  such  things  as:  what  results  were  needed  by  NPMOD 
and  NSAWC  personnel;  what  data  could  be  collected  and  what  results  could  be 
obtained  from  the  data;  how  to  collect  the  data  without  imposing  an  excessive 
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additional  work  on  NPMOD  or  NSAWC  personnel;  and  how  to  present  the  results 
in  the  most  useful  forms  for  NPMOD  and  NSAWC  personnel.  In  addition, 
significant  IT  challenges  had  to  be  overcome  to  make  the  system;  user  friendly; 
flexible  to  accommodate  user  requested  changes;  as  automated  as  possible  so 
the  work  load  on  NPS  personnel  is  minimized;  capable  of  providing  near  real 
time  results;  accurate;  and  robust. 
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III.  RESULTS 


A.  OVERVIEW 

In  this  chapter,  we  present  the  results  from  our  analyses  of  the  data 
provided  by  the  USS  Saipan  METOC  unit  and  by  NPMOD  (see  NPS  METOC 
Metrics  Support  Site,  http://wx.met.nps.navy.mil/metrics/index.html).  We  also 
present  our  assessment  of  the  IT  system  we  developed  for  collecting  and 
analyzing  this  data.  In  this  chapter,  we  focus  on  results  that  help  demonstrate 
and  validate  the  concept  and  the  implementation  of  the  IT  system,  including  the 
IT,  METOC,  and  operational  issues  addressed  by  the  system.  As  a  secondary 
focus,  we  also  address  in  this  chapter  the  results  of  our  analyses  of  the  METOC 
and  operational  data  collected  as  part  of  the  testing  and  initial  implementation  of 
the  system. 

B.  USS  SAIPAN 

The  daily  stoplight  forecasts  provided  by  the  USS  Saipan  METOC  unit  for 
September  and  October  2004  were  analyzed,  and  FAC,  POD,  and  NAF  were 
calculated  for  the  five  different  operations  supported  by  these  forecasts  and  for 
the  three  different  forecast  categories  (see  Chapter  II). 

1.  Saipan  Forecast  Performance  Metrics 

The  three  forecast  performance  metrics  used  in  this  portion  of  the  study, 
FAC,  POD,  and  NAF,  were  plotted  against  the  forecast  lead  time  at  six  hour 
intervals.  These  metrics  are  useful  in  assessing  forecast  performance,  but  in  this 
case  their  use  is  limited  by  the  small  number  of  forecasts  in  the  data  set.  For 
example,  POD  for  aviation  plot  (Figure  22)  shows  that  the  probability  of  detecting 
a  yellow  event  is  100%  at  a  lead  time  of  12  hours  and  28.6%  at  a  lead  time  of  18 
hours.  However,  these  results  are  based  on  only  three  yellow  events  at  12 
hours  and  seven  events  at  18  hours,  so  they  may  be  misleading, 
a.  FAC 

The  FAC  graphs  for  the  different  Saipan  operation  types  and 
forecast  categories  are  shown  in  Figures  17-21.  Some  of  these  figures  show 
zero  FAC  values  for  some  forecast  categories.  These  zero  values  indicate  in 
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general  that  no  forecasts  of  those  categories  were  issued  for  the  specified 
operation  type  and  lead  time.  This  is  primarily  a  result  of  the  small  number  of 
forecasts  in  the  data  set.  The  total  FAC  (mean  of  red,  yellow,  and  green  FAC)  is 
probably  the  most  useful  quantity  in  these  figures,  given  the  small  number  of 
forecasts  being  analyzed.  Figures  17-21  indicate  that  the  total  FAC  values  for 
each  operation  type  were  primarily  driven  by  one  event  category.  For  example, 
the  total  FAC  for  aviation  forecasts  (Figure  17)  was  driven  by  the  forecasts  of 
green  events  because  these  were  the  most  numerous  events.  Similarly,  total 
FAC  values  for  LCAC  and  LCU  operations  were  primarily  determined  by  red 
events,  while  yellow  and  green  events  occurred  much  less  frequently  and  had 
very  little  impact  in  the  total  FAC  values.  Note  that  the  total  FAC  was  about  40- 
80%  for  all  operation  types.  These  values  are  consistent  with  those  found  by 
Hinz  (2004)  and  Jarry  (2005)  for  other  METOC  and  AFW  forecasts.  The  total 
FAC  decreases  with  increasing  lead  time  for  aviation  forecasts  (Figure  17),  but 
changes  little  or  increases  with  lead  time  for  the  four  other  operation  types 
(Figures  18-21).  This  wide  variation  in  the  relationship  between  FAC  and  lead 
time  for  different  operation  types  was  also  found  by  Hinz  (2004)  in  his  analyses 
of  OIF  forecasts.  Hinz  (2004)  provides  some  possible  explanations  for  increases 
in  FAC  with  increasing  lead  time  (e.g.,  greater  reliance  at  short  lead  times  on 
problematic  mesoscale  models,  greater  model  adjustment  problems  at  short  lead 
times).  These  explanations  may  apply  here  but  with  such  a  small  sample  size, 
the  reasons  are  difficult  to  determine.  The  FAC  values  show  a  marked  24  hour 
periodicity  due  to  persistence  in  forecasts  (e.g.,  forecast  errors  that  persist  over 
several  lead  times)  and  the  daily  sampling  of  forecast  and  observational  data 
used  for  our  analyses. 

b.  POD 

The  POD  graphs  for  the  different  Saipan  operation  types  and 
forecast  categories  are  shown  in  Figures  22-26.  These  figures  are  similar  to 
those  for  FAC  (Figures  17-21),  for  example  in  the  interpretation  of  the  zero 
values,  the  similarities  to  the  results  of  Hinz  (2004)  and  Jarry  (2005),  etc.  As  with 
FAC,  the  total  POD  values  for  each  operation  type  were  primarily  driven  by  one 
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type  of  event.  For  example,  the  total  POD  for  aviation  forecasts  (Figure  22)  was 
driven  by  the  forecasts  of  green  events  because  these  were  the  most  numerous 
events.  Similarly,  total  POD  values  for  LCAC  and  LCU  operations  were  primarily 
determined  by  red  events,  while  yellow  and  green  events  occurred  much  less 
frequently  and  had  very  little  impact  in  the  overall  values. 

2.  Saipan  Data  Collection  and  Analysis  System 
Overall,  the  system  that  was  developed  to  allow  for  the  collection  of  the 
forecast  and  observation  data  performed  well  and  was  brought  online  relatively 
rapidly.  The  main  shortcoming  of  the  system  was  in  the  analysis  and  delivery  of 
the  metrics  report  to  the  Saipan  METOC  team.  There  was  a  delay  of 
approximately  three  months  from  the  time  the  data  was  entered  until  the  final 
report  was  delivered  to  the  Saipan.  Much  of  this  delay  came  from  starting  to 
work  with  the  Saipan  prior  to  having  a  fully  functional  and  well  tested  system. 
However,  some  of  this  delay  is  inherent  in  the  fully  functional  system  used  for 
analyzing  and  reporting  on  the  Saipan  data.  The  lessons  we  learned  from 
dealing  with  these  inherent  problems  led  us  to  develop  a  faster  and  more 
automated  system  when  we  began  work  with  NPMOD. 

a.  Data  Collection  and  Archiving 

For  the  Saipan  system,  we  began  work  on  the  data  collection 
portion  of  the  system  in  December  2004.  Data  collection  was  made  operational 
on  the  NPS  web  server  by  early  March  2005.  The  system  was  expanded  by  the 
end  of  March  to  include  the  ability  to  review  and  delete  the  data  as  necessary. 

The  system  took  the  Saipan’s  hourly  observations  stored  in  the 
database  and  converted  them  to  stoplight  forecast  categories  (red,  yellow,  or 
green)  using  computer  scripts  written  in  PERL.  These  converted  observations 
and  the  original  forecasts  were  then  exported  into  Excel  spreadsheets  for 
analysis  and  display.  The  Excel  analyses  included:  (a)  comparisons  of  the 
forecasted  categories  to  the  observed  categories;  and  (b)  calculation  of  the 
forecast  performance  metrics  using  formulas  and  macros  designed  and  built  into 
Excel. 
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In  the  original  system  developed  for  the  Saipan,  a  PERL  program 
was  developed  that  automatically  retrieved  and  decoded  the  hourly  weather 
observations  and  then  stored  the  decoded  information  into  the  main  database. 
Regrettably,  we  were  unable  to  use  this  part  of  the  system  because  the  hourly 
observations  archived  by  the  Saipan  were  in  a  nonstandard  format. 

The  results  of  the  Saipan  portion  of  the  study  moved  our  overall 
project  much  closer  to  our  main  goal  of  developing  an  automatic  collection  and 
archiving  system,  with  near  real  time  analyses.  In  particular,  our  Saipan 
experiences  demonstrated  some  key  shortfalls  in  our  approach  that  were  later 
corrected  with  the  design  of  the  system  for  NPMOD. 

C.  NPMOD  RESULTS 

The  results  of  our  metrics  support  for  NPMOD  can  be  divided  into  two 
categories,  metrics  results  and  computer  system  results.  The  metrics  results  can 
be  subdivided  into  four  areas: 

•  Operational  impacts  metrics 

•  Planning  forecast  and  operational  performance  metrics 

•  Weather  phenomena  metrics 

•  Air  field  forecast  performance  metrics 

TAWS  and  TAWS  related  data  was  also  collected  during  this  study  but 
only  for  a  small  number  of  missions.  Due  to  the  small  amount  of  this  data,  we  did 
not  analyze  it  in  this  study.  This  data  will  continue  to  be  collected  for  future  air 
wing  deployments  and  will  be  analyzed  in  future  studies. 

1.  Metrics  Results  -  Operational  Impacts  Metrics 

The  review  of  our  operational  impacts  metrics  results  is  divided  into  three 
categories:  missions  and  weapons  saved,  initial  team  planning  changes,  and 
tactical  impacts  during  execution. 

a.  Missions  and  Weapons  Saved 

The  changes  to  mission  plans  made  by  NSWAC  planners  in 
response  to  the  planning  forecasts  provided  by  NPMOD  were  used  to  determine 
the  number  and  percent  of  missions  saved,  weapons  saved,  and  missions  and 
weapons  potentially  saved  (see  Chapter  II,  section  3. a  for  an  explanation  of 
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these  metrics).  Of  all  missions  in  our  data  set,  10  %  were  saved,  8  %  of 
weapons  employed  on  separate  missions  were  saved,  and  18  %  of  missions  and 
weapons  were  potentially  saved  (Figure  27).  These  results  indicate  that  in  18  % 
of  the  missions,  the  planning  forecasts  led  to  an  increase  in  NSAWC’s 
operational  performance,  and  that  these  forecasts  could  have  done  so  in  an 
additional  18  %  of  missions.  These  increases  in  operational  performance  are,  or 
could  have  been,  improvements  in  scheduling,  safety,  efficiency,  effectiveness 
through  the  use  of  planning  forecasts  to  avoid  negative  weather  impacts.  The 
missions  and  weapons  saved  results  are  summarized  in  Table  1 . 

Table  1.  Summary  of  missions  saved,  weapons  saved  and  missions  and 


weapons  potentially  saved  at  NSAWC,  Fallon. 


Missions 

Number 

Percent 

Missions  Saved 

5 

10 

Weapons  Saved 

4 

8 

Missions  &  Weapons 

Potentially  Saved 

9 

18 

b.  Initial  Team  Planning  Changes 

We  also  analyzed  the  changes  made  by  NSAWC  planners  to 
determine  the  frequency  of  weather  related  changes,  and  the  relationships 
between  the  types  of  change  and  specific  weather  phenomena,  In  76%  of  all 
missions,  no  change  was  made  to  mission  as  a  result  of  weather  forecast.  Of  the 
24%  that  required  a  change  in  response  to  weather,  most  (50%,  or  12  %  of  all 
missions),  were  changed  due  to  visibility  and  or  ceiling  restrictions  that  would 
have  negatively  impacted  the  weapons  that  had  been  planned  for  the  missions. 
The  complete  breakdown  of  initial  team  planning  changes  is  shown  in  Table  2 
and  Figure  28. 
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Table  2.  Summary  of  number  and  percent  of  missions  which  resulted  in  a 


team  planning  change  due  to  forecasted  weather. 


No  Change 
in  Mission 

Mission 

Delayed  or 

Rescheduled 

Weapons 

Changed 

Due  to 

Wind 

Weapons 

Changed 

Due  to 

Visibility 

Other 

Changes 

Number 

37 

5 

1 

6 

0 

Percent 

76 

10 

2 

12 

0 

c.  Tactical  Impacts  During  Execution 

A  number  of  missions  were  planned  despite  forecast  of  negative 
weather  impacts.  For  these  missions,  NSWAC  personnel  flew  pre-mission  flights 
within  a  few  hours  of  mission  takeoff  to  evaluate  the  weather  conditions  and 
make  last  minute  changes  to  mission  plans  in  response  to  the  weather  conditions 
observed  during  the  pre-mission  flights.  These  included  changes  such  as: 
mission  delay,  change  in  tactics,  change  in  flight  levels,  reduction  in  the  scope  of 
the  mission,  and  mission  cancellation.  We  analyzed  these  mission  changes  to 
determine  the  frequency  of  the  specific  changes  and  to  later  relate  these 
changes  to  forecast  accuracy  and  specific  weather  phenomena.  The  most 
common  change  (18%  of  missions)  was  a  switch  to  a  high  war  mission  (increase 
in  flight  altitude),  with  the  next  most  common  change  (14%  of  missions)  being 
cancellation.  Table  3  and  Figure  29  summarizes  the  results  for  all  changes 
made  in  response  to  pre-mission  weather  observations. 
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Table  3.  Summary  of  the  number  and  percent  of  tactical  mission  impacts 


that  resulted  from  weather  during  the  missions. 


Missions 

Number 

Percent 

No  Impact 

23 

47 

Delay  of  Mission 

2 

4 

Cancellation  of  Mission 

7 

14 

Other  -  Schedule 

0 

0 

Low  War 

2 

4 

High  War 

9 

18 

Split  War 

5 

10 

Other  -  War 

2 

4 

Partial  No  Strike 

2 

4 

Partial  No  Bombs 

4 

8 

Partial  Missed 

0 

0 

Other  -  Partial 

0 

0 

2.  Metrics  Results  -  Planning  Forecast  and  Operational 
Performance  Metrics 

To  determine  the  effectiveness  of  the  weather  support  provided  during  the 
mission  planning  phase,  we  compared  information  about  planning  forecast 
accuracy  with  information  about  planning  changes.  These  comparisons  allowed 
us  to  identify  relationships  between  forecast  performance  and  operational 
performance.  For  example,  they  allowed  us  to  determine  cases  in  which  an 
accurate  forecast  of  negative  weather  impacts  was  issued  but  planners  did  not 
alter  their  plans  to  account  for  negative  weather  conditions,  and  mission 
performance  was  reduced  because  of  accurately  forecasted  negative  weather 
conditions.  These  results  are  summarized  in  Table  4. 
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Table  4.  Summary  of  the  relationships  between  planning  forecasts  and 
_ planning  changes  by  number  and  percent  of  missions. _ 


Missions 

Number 

Percent 

No  change  made  during  team  planning 

27 

55 

Weather  was  forecasted  correctly  and  led  to  a 
successful  mission  change 

9 

18 

Weather  was  forecasted  correctly  but  mission 
planner  made  no  change 

9 

18 

Weather  forecast  was  incorrect  and  led  to  an 
unnecessary  mission  change 

2 

4 

Weather  forecast  was  incorrect,  and  a  correct 
forecast  would,  if  accepted  by  planners,  have  led  to  a 
necessary  mission  change 

0 

0 

Not  enough  information  to  know  if  decision  was  good 
or  bad 

1 

2 

None  of  the  above 

1 

2 

To  provide  more  insight  into  the  weather  conditions  that  led  to  negative 
weather  forecasts  and  actual  negative  weather  impacts,  we  related  specific 
weather  phenomena  to  the  missions  that  fell  into  categories  2-5  in  Table  4.  For 
example,  the  nine  missions  in  which  “Weather  was  forecasted  correctly  and  led 
to  a  successful  mission  change”  were  categorized  according  to  their  associated 
negative  weather  phenomena.  This  led,  for  this  example,  to  the  determination 
that  for  all  nine  of  these  missions,  negative  impacts  due  to  cloud  layers  and  cloud 
thickness  were  accurately  forecasted  and  planners  made  necessary  changes  in 
response  to  those  forecasts.  The  results  of  these  analyses  are  summarized  in 
Table  5. 
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Table  5.  Summary  of  the  relationships  between  planning  forecasts  and 


planning  changes  by  weather  phenomena  forecasted  to  have 
negative  mission  impacts. _ _ _ _ _ 


Surface 

Wind 

Aloft 

Wind 

Altitude 

Clouds 

Visibility 

Weather  was  forecasted 
correctly  and  ed  to  a 
successful  mission 
change 

Number 

5 

1 

5 

9 

7 

Percent 

56 

11 

56 

100 

78 

Weather  was  forecasted 
correctly  but  mission 
planner  made  no  change 

Number 

0 

0 

1 

7 

2 

Percent 

0 

0 

11 

78 

22 

Weather  forecast  was 
incorrect  and  led  to  an 
unnecessary  change. 

Number 

2 

2 

1 

1 

1 

Percent 

100 

100 

50 

50 

50 

Weather  forecast  was 
incorrect,  and  a  correct 
forecast  would  have  led 
to  a  necessary  mission 
change 

Number 

0 

0 

0 

0 

0 

Percent 

0 

0 

0 

0 

0 

3.  Metrics  Results  -  Weather  Phenomena  Metrics 

Analyses  were  also  conducted  to  identify  the  weather  phenomena  that 
were  forecasted  to  cause  negative  impacts  and  that  actually  did  cause  negative 
impacts.  The  weather  phenomenon  that  was  most  commonly  forecasted  to 
cause  negative  impacts,  and  that  actually  did  so,  was  cloud  layer  thickness.  This 
phenomenon  was  forecasted  to  impact  41%  of  the  missions  and  actually 
impacted  27%  of  them.  The  next  most  common  phenomenon  was  reduced 
surface  visibility  due  to  fog,  haze,  dust  storm,  or  precipitation.  Reduced  surface 
visibility  was  forecasted  to  impact  22%  of  all  missions  and  actually  impacted 
14%.  The  complete  breakdown  by  phenomenon  of  forecasted  negative  impacts 
is  shown  in  Table  6  and  Figure  30.  The  corresponding  results  for  observed 
negative  impacts  are  shown  in  Table  7  and  Figure  31 . 
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Table  6.  Summary  of  the  number  and  percent  of  missions  forecasted  to  be 
_ impacted  by  specific  weather  phenomena.  _ _ 


Forecasted 

Surface 

Aloft 

Altitude 

Cloud 

Surface 

Other 

Winds 

Winds 

Restrictions 

Thickness 

Visibility 

Number 

8 

4 

8 

20 

11 

1 

Percent  of 

Missions 

16 

8 

16 

41 

22 

2 

Table  7.  Summary  of  the  number  and  percent  of  missions  observed  to  be 
_ impacted  by  specific  weather  phenomena.  _ _ 


Observed 

Surface 

Aloft 

Altitude 

Cloud 

Surface 

Other 

Winds 

Winds 

Restrictions 

Thickness 

Visibility 

Number 

5 

0 

4 

13 

7 

1 

Percent  of 

Missions 

10 

0 

8 

27 

14 

2 

4.  Metrics  Results  -  Air  Field  Forecast  Performance  Metrics 

We  calculated  FAC  and  POD  for  the  NPMOD  forecasts  for  the  NAS  Fallon 
air  field.  Observational  data  from  an  automated  observing  system,  and  from 
NPMOD  personnel  was  available  for  this  location.  FAC  and  POD  were 
calculated  for  each  of  three  weather  parameters:  sustained  winds,  minimum 
ceiling,  and  visibility.  The  results  are  shown  in  Table  8. 


Table  8.  Forecast  accuracy  and  probability  of  detection  results  for  the  NAS 


Fallon  air  field  for  each  of  three  forecasted  weatl 

ner  phenomena. 

Wind 

Ceiling 

Visibility 

FAC  (%) 

92 

82 

100 

POD  (%) 

100 

68 

0 

5.  Computer  System  Results 

The  web-based  system  developed  in  support  of  NPMOD  is  based  on  the 
system  developed  to  support  USS  Saipan,  but  has  three  main  improvements:  (a) 
the  automation  of  data  collection  and  analysis;  (b)  the  on-demand,  or  near  real- 
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time,  results;  and  (c)  a  web-based  output  report  tailored  to  meet  the  specific 
requirements  and  goals  of  NPMOD.  The  development  of  the  NPMOD  system 
required  a  tremendous  number  of  web  pages  and  computer  scripts,  over  35  for 
the  output  reports  alone.  An  example  of  the  computer  code  use  to  produce  an 
output  report  is  provided  in  Appendix  B.  Despite  the  complexity  of  the  design 
and  development  of  the  system,  it  has  proven  to  be  very  robust,  has  required 
little  additional  redesign,  and  has  been  very  positively  reviewed  by  users  at 
NPMOD. 

However,  several  improvements  to  the  system  were  requested  by  users 
once  the  system  was  made  operational  at  NPMOD.  One  improvement  would  be 
to  increase  the  query  capability  of  the  web  interface  to  give  users  more 
capabilities  in  searching  the  archived  data.  Presently  users  can  only  sort  the 
data  by  individual  air  wings  or  by  all  air  wings.  An  expanded  search  function 
would  allow  users  to  sort  results  by  month  or  other  time  frames  and  other  more 
specific  searches.  An  additional  recommendation  is  to  give  the  user  the  ability  to 
edit  the  data  associated  with  a  specific  mission.  Currently  the  program  only 
allows  for  the  review  and  deletion  of  data  but  does  not  permit  editing  of  data.  We 
expect  that  these  improvements  will  be  made  in  the  near  future,  after  the 
completion  of  this  study. 

a.  On  Demand  Results 

Unlike  the  USS  Saipan  data  analyses,  which  was  done  offline  using 
Excel  spreadsheets,  the  analyses  of  NPMOD  data  are  incorporated  directly  into 
the  user  web-interface.  By  embedding  PHP  scripts  that  perform  the  analyses 
and  deliver  the  results  directly  in  the  output  web  page,  users  are  able  to  view  the 
results  immediately  after  the  input  data  has  been  entered.  The  online  data  entry, 
automated  online  data  analyses,  and  automated  near  real  time  reporting  of 
results  allows  the  users  of  the  system  to  be  relatively  independent  of  the  system 
designers.  This  independence  is  a  critical  for  operationally  implementing  the 
system  with  other  METOC  and  AFW  units. 
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b.  Tailored  Output  Reports 

Through  discussions  with  NPMOD  (Cantu  2005),  an  output  report 
was  designed  to  provide  them  with  the  specific  metrics  content  and  format  that 
were  needed  by  NPMOD  managers  and  its  NSAWC  customers.  This  tailored 
report  was  developed  and  is  used  to  provide  feedback  on  NPMOD  and  NSAWC 
performance,  to  detail  operational  impacts  of  support  provided,  and  to  allow  for 
metrics  results  to  be  included  in  briefings  to  newly  arriving  air  wings  and  their 
METOC  support  teams.  The  reports  can  be  generated  at  the  system  web  site  for 
various  combinations  of  data  (e.g.,  reports  for  individual  air  wings,  for 
combinations  of  air  wings,  or  for  individual  months).  A  sample  output  report  is 
shown  in  Figure  32. 
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IV.  SUMMARY,  DISCUSSIONS  AND  RECOMMENDATIONS 


A.  SUMMARY  OF  PROCEDURES  AND  RESULTS 

This  study  successfully  developed,  tested  and  implemented  a  web-based 
system  for  collecting  and  analyzing  forecasts  and  observations  to  determine 
forecast  performance  and  the  operational  impacts  of  weather  support  (see  NPS 
METOC  Metrics  Support  Site,  http://wx.met.nps.navy.mil/~mdbutler/index.html). 
This  system  provides  near  real-time  or  on-demand  results  for  the  end  user. 
While  several  Naval  units  participated  in  this  study,  the  main  focus  was  NPMOD 
which  provides  weather  support  to  the  NSAWC.  The  implementation  of  the 
system  allows  NPMOD  personnel  to  enter  data  remotely  and  view  their  results 
on-demand  via  the  internet.  The  system  has  become  an  integral  part  of  the 
planning  and  execution  cycle  at  NMPOD  (Cantu  2005). 

We  applied  several  forecast  performance  and  operational  impacts  metrics 
used  by  Hinz  (2004)  and  Jarry  (2005),  and  also  developed  several  new  metrics. 
The  metrics  used  in  this  study  allowed  us  to  quantitatively  assess: 

1 .  performance  of  the  forecasts  used  in  planning  NSAWC  missions  (e.g. 
forecast  accuracy,  probability  of  detection) 

2.  impacts  of  forecasts  on  mission  planning  (e.g.  changes  in  mission 
schedule,  targets,  weapons,  tactics) 

3.  deviations  from  mission  plans  that  occurred  during  missions  in 
response  to  weather  conditions  actually  encountered  by  air  crews  (e.g. 
changes  in  tactics,  targets,  weapons  use) 

4.  positive  and  negative  impacts  on  mission  planning,  execution,  and 
outcomes  due  to  forecasts  (e.g.  missions  that  avoided  or  incurred 
delays,  cancellations,  inappropriate  weapons  load  outs,  missions  that 
might  have  avoided  problems  had  the  forecast  been  followed  by 
mission  planners) 

5.  METOC  tactical  decision  aid  forecast  accuracy  and  mission  impacts 
(e.g.,  TAWS  WOF  accuracy,  weather  impacts  on  weapon  sensors) 
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6.  forecast  performance  and  mission  impacts  with  respect  to  specific 
weather  factors  (e.g.,  surface  and  aloft  winds,  dust,  fog) 

B.  ADDITIONAL  WORK  COMPLETED 

In  addition  to  the  study  described  in  the  preceding  sections,  we  also 
worked  on  several  related  but  less  extensive  studies.  One  of  the  studies  was  in 
support  of  a  request  from  the  METOC  team  onboard  the  USS  Nimitz.  This  team 
wanted  assistance  in  developing  an  automated  system  for  collecting  and 
analyzing  its  air  operations  brief  which  was  given  several  times  a  day.  A  basic 
system  was  developed  through  collaboration  between  the  USS  Nimitz  and  NPS 
to  allow  for  the  collection  of  the  brief,  but  the  complete  system  to  archive  hourly 
observations  and  perform  analyses  has  not  yet  been  implemented  due  to  time 
constraints. 

Preliminary  work  was  also  completed  on  a  system  to  allow  for  the 
automated  collection  of  both  TAFs  and  hourly  observations  for  U.S.  Naval  air 
facilities,  with  the  goal  of  using  this  data  to  perform  automated  TAF  verification. 
The  ultimate  goal  of  this  work  was  to  develop  a  system  that  would  allow  for  the 
automated  TAF  verification  for  all  U.S.  Navy  air  facilities  and  local  forecast  and 
warning  verification  for  all  U.S.  Navy  shore  facilities.  The  data  collection  and 
archiving  part  of  the  system  was  created,  but  the  verification  part  of  the  system 
remained  in  development  due  to  time  constraints. 

Finally,  at  the  request  of  the  commanding  officer  at  Strike  Group 
Oceanography  Team  Norfolk,  CDR  Steve  Woll,  software  was  installed  at  NPS  on 
the  SIPRNET  to  allow  for  the  porting  of  metrics  systems  to  the  classified  side. 
This  was  a  first  step  in  adapting  the  automated  online  metrics  system  described 
in  the  preceding  system  for  use  in  a  classified  setting. 

C.  RECOMMENDATIONS 

The  web-based  system  that  was  developed  for  this  study  is  in  operational 
used  by  NPMOD  personnel  in  support  of  NSAWC  missions.  The  missions  being 
supported  for  this  study  are  being  conducted  by  Navy  strike  air  wings  preparing 
for  upcoming  deployments.  Also  at  this  time,  METOC  forecasters  are  being 
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deployed  to  accompany  the  air  wings  as  they  train  at  NAS  Fallon  (Cantu  2005). 
Eventually,  these  same  forecasters  will  join  the  air  wings  as  they  are  forward 
deployed.  One  of  key  recommendations  for  this  study  is  to  familiarize  the 
deploying  forecasters  with  this  system  so  that  data  from  real  world  naval 
missions  can  be  collected  as  air  wings  and  their  forecasters  deploy  together. 

A  second  recommendation  is  to  adapt  the  web-based  system  to  other 
types  of  military  weather  support.  For  example,  the  current  system  could  be 
easily  adapted  to  collect  and  analyze  the  performance  of  the  stoplight  or  Go/No 
Go  forecasts  produced  by  many  METOC  and  AFW  units.  This  would  allow  many 
units  to  have  their  forecasts  analyzed  and  reported  on  in  near  real-time  with 
relatively  little  effort  by  unit  personnel.  Presently,  many  units  do  not  even  attempt 
to  evaluate  their  forecasts  because  they  perceive  the  learning  and  development 
curve  to  be  very  steep,  and  the  implementation  costs  very  high.  The  system 
developed  in  this  study  greatly  reduces  the  knowledge  and  skills  required  of 
users  by  automating  the  main  steps  needed  to  conduct  forecast  performance 
assessments. 

A  major  feature  of  the  system  we  have  developed  is  its  ability  to  assess 
the  operational  impacts  of  METOC  support.  As  far  as  we  have  been  able  to 
determine,  there  is  no  comparable  process,  especially  such  an  automated  near 
real  time  process,  in  existence  any  other  civilian  or  military  organization.  Thus, 
we  recommend  the  adaptation  of  this  system  to  other  military  and  civilian 
organizations  for  the  purpose  of  assessing  how  operations  are  affected  by 
METOC  conditions  and  METOC  forecasts.  A  prime  candidate  for  adaptation  of 
the  system  is  air  combat  units  and  Air  Force  combat  weather  teams  at  Nellis  Air 
Force  Base,  Nevada,  where  strike  training  similar  to  that  at  NAS  Fallon  is 
conducted.  With  minor  adjustments  to  the  data  being  collected,  similar 
operational  impacts  and  forecast  accuracies  reports  could  be  produced  for  USAF 
missions. 

We  also  recommend  training  permanent  personnel  to  continue  to  develop 
and  maintain  the  current  system.  One  of  the  major  concerns  of  the  NPMOD 
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personnel  was  that  the  system  that  was  developed  for  this  study  would  no  longer 
be  available  after  the  completion  of  the  study.  While  the  system  was  designed  to 
allow  for  continued  use  after  the  study  was  completed,  additional  work  will  be 
required  to  maintain  or  expand  the  system  to  other  units.  The  continuity  of 
operations  provided  by  permanent  personnel  would  be  pivotal  to  the  continuing 
success  of  this  program. 

Finally,  we  recommend  centralizing  research,  development,  and  testing  of 
forecast  performance  and  operational  impacts  metrics  systems  at  NPS.  This 
would  enhance  the  standardization  of  data,  analysis  procedures,  results,  and 
reports.  It  would  also  promote  efficiency  by  having  faculty,  researchers,  and 
METOC  and  AFW  officers  at  NPS  to  focus  on  research  and  development,  and 
allowing  operational  METOC  and  AFW  units  to  focus  on  providing  customer 
support.  Additionally,  due  to  Department  of  Defense  Information  Technology  21 
(IT21)  restrictions,  much  of  the  software  used  in  the  NPS  automated  online 
system  work  is  unavailable  to  most  military  units.  Alternative  software  that  is 
available  to  these  units  is  much  less  suitable  for  such  systems. 

D.  FUTURE  WORK 

After  the  successful  implementation  of  the  system  several  additional  units 
were  interested  in  participating  in  the  study,  but  due  to  time  constraints  the  study 
was  unable  to  fully  accommodate  these  requests.  These  willing  participants 
provide  excellent  opportunities  to  expand  the  current  system  and  develop  a  fleet 
wide  METOC  metrics  program.  We  expect  that  these  opportunities  will  be 
pursued  through  additional  collaborations  between  NPS  and  METOC  and  AFW 
units.  One  such  additional  collaborative  project  is  currently  underway.  In  this 
project,  NPS  will  work  with  U.S.  Air  Force  combat  weather  teams  (CWTs)  in  the 
Air  Combat  Command  to  adapt  the  system  employed  at  NAS  Fallon  for  use  by 
the  CWTs  and  their  customers. 


42 


LIST  OF  REFERENCES 


AFI  15-114,  2001.  Functional  Resource  and  Weather  Technical  Performance 
Evaluation,  Air  Force  Instruction  15-114.  HQ  USAF/XOW.  [Available  on-line 
at  http://www.e-publishinq.af.mil/  ]  Sept  05. 

Cantu,  A.,  2001 :  The  Role  of  Weather  in  Class  A  Naval  Aviation  Mishaps.  M.  S. 

Thesis,  Naval  Postgraduate  School,  Monterey,  CA,  86pp. 

Cantu,  A.,  2004.  Personnel  communications  with  LCDR  Cantu,  Officer  in 
Charge,  NPMOD,  Fallon,  April  2004  -  present. 

Clark,  V.  ADM,  CNO  2004:  Chief  of  Naval  Operations,  CNO,  Guidance  for  2004. 
[Available  on-line  at  http://www.chinfo.navy.mil/navpalib/cno/clark- 
quidance2004.html  ]  Sept  05. 

Hinz,  J.,  2002:  Fleet  Cost  Savings  Attributable  to  OTSR  /  WEAX  Services. 

NLMOC  Report,  [N/A  online,  contact  murphree@nps.edu  to  request  a  copy] 

Hinz,  J.,  2004:  Developing  and  Applying  METOC  Metrics  to  Sea  Strike 
Operations:  A  Case  Study  of  Operations  Iraqi  Freedom.  M.  S.  Thesis,  Naval 
Postgraduate  School,  Monterey,  CA,  195pp. 

Jarry,  J.  2005:  Analysis  of  Air  Mobility  Command  Weather  Mission  Execution 
Forecasts:  Metrics  of  Forecast  Performance  and  Impacts  on  War  Fighting 
Operations.  M.  S.  Thesis,  Naval  Postgraduate  School,  Monterey,  CA,  210pp. 

Martin,  B.,  2002:  METOC  and  Naval  Afloat  Operations:  Risk  Management, 
Safety,  and  Readiness.  M.  S.  Thesis,  Naval  Postgraduate  School,  Monterey, 
CA,  72pp. 

Tomaszeski,  S.  RADM  2005:  N7C  Status  Report  on  the  Naval  Oceanography 
Program,  [N/A  online,  contact  murphree@nps.edu  to  request  a  copy] 


43 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


44 


APPENDIX  A.  FIGURES 
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Figure  1 .  Example  of  USS  Saipan  daily  96  hour  forecast  for  aviation,  LCAC, 
LCU,  smallboat,  and  RAS  operations. 
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Figure  2.  NPS  METOC  Metrics  Support  Site  home  page  (available  at 
http://wx.met.nps.navy.mil/~mdbutler/index.html). 
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Figure  3.  USS  Saipan  data  collection  page  (viewable  at 

http://wx.met.nps.navy.mil/~mdbutler/saipan/enter/display_forecast. 

html). 
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NPMOD  FALLON 
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personnel  via 
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Data  written  to  MYSQL 
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Data  analyzed  per  NPS 
metrics  procedures 


Data  entered  remotely  via 
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personnel  ^ 


NPMOD  FALLON 


Figure  4.  Graphical  depiction  of  the  flow  of  data  collected  and  analyzed  for 
NPMOD  Fallon,  and  of  reports  issued  to  NPMOD  Fallon. 


3-4  Days  Prior 


Figure  5.  Input  from  NPMOD  Fallon  METOC  into  the  NSAWC  planning  and 
execution  cycle,  and  from  NSAWC  to  NPMOD  Fallon  after  mssion 
execution. 
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Figure  6.  Nine  main  categories  of  data  collected  by  NPMOD  Fallon  for 
analysis  of  NPMOD  Fallon  support  of  NSAWC  operations. 


1.  Event  Data 
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Figure  7.  Flight  and  mission  information  portion  of  online  collection  form. 


2.  Intial  Team  Planning  Forecasted  Negative  Weather  Impacts  (Select  all  that  apply) 

(Check  all  parameters  that  are  forecasted  to  impact  event  negatively) 

®no  Oyes 

Excessive  Surface  Winds 

®no  Oyes 

Excessive  Winds  Aloft  up  to  30  Kft 

®no  Oyes 

Altitude  Restrictions  due  to  N/A 

®no  Oyes 

Cloud  Layers/Thickness 

®no  Oyes 

Reduced  Surface  Visibility  due  to  n/a 

®no  Oyes 

Low  Thermal  Contrast 

®no  Oyes 

Other  event  forecasted.  Please  explain:  ! 

-igure  8.  Forecasted  negative  impacts  portion  of  online  collection  form. 
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4a.  Change  in  schedule 

®no  Cyes  Event  Delayed/Reschedule 
©no  Oyes  Entire  event  canceled 

©no  ©yes  Other  change  in  schedule  Please  enter  other  in  box  provided 

4b.  Type  of  war 

©no  Oyes  Low  war 
©no  Oyes  High  war 
©no  Oyes  Split  war 
©no  Oyes  Normal  war 

©no  Oyes  Other  type  of  war  Please  enter  other  In  box  provided 

4c.  Partial  Mission 

©no  Oyes  Non  strike  aircraft  did  not  complete  mission 
©  no  Oyes  Bomber  did  not  drop 
©no  Oyes  Bombs  missed  target 

©no  Oyes  Other  partial  mission  Please  enter  other  in  box  provided 


Figure  1 1 .  Negative  impacts  from  weather  portion  of  online  collection  form. 
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5.  Impacting  Weather 

(Check  all  weather  parameters  that  impacted  the  event  negatively) 
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Excessive  Surface  Winds 

©no  Oyes 

Excessive  Winds  Aloft  up  to  30  Kft 
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Reduced  Surface  Visibility  due  to  n/a 
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Low  Thermal  Contrast 

©no  Oyes 

Other  event  forecasted.  Please  explain: 

Figure  12.  Negatively  impacting  weather  phenomena  portion  of  the  online 
collection  form. 
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Figure  13.  Local  airfield  forecast  verification  portion  of  the  online  collection 
form. 


8.  TAWS  Weather  Impact 
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because  of  the  actual  weather  forecasted) 

©no  Oyes  No  changes  required.  There  is  no  forecast  impact  to  the  sensor 

©no  Oyes  No  changes  required.  Possible  IR  Sensor  impact  for  visibility/ceiling  noted  earlier. 

©no  Oyes  Mission  change  for  thermal  crossover 

©no  Oyes  Attack  heading  change  perTAWS  ranges  prediction 

©no  Oyes  Other:  [ _ 0 


:igure  14.  TAWS  effectiveness  portion  of  the  online  collection  form. 
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Figure  15.  Total  Forecast  Accuracy  (FAC)  for  five  operation  types  for  USS 
Saipan,  September  -  October  2004. 
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Figure  16.  Total  Probability  of  Detection  (POD)  for  five  operation  types  for 
USS  Saipan,  September  -  October  2004. 
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Figure  17.  Forecast  Accuracy  (FAC),  USS  Saipan,  Aviation  Forecasts, 
September  -  October  2004. 
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Figure  18.  Forecast  Accuracy  (FAC),  USS  Saipan,  LCAC  Forecasts, 
September  -  October  2004. 
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Figure  19.  Forecast  Accuracy  (FAC),  USS  Saipan,  LCU  Forecasts,  September 
-  October  2004. 
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Figure  20.  Forecast  Accuracy  (FAC),  USS  Saipan,  Smallboat  Forecasts, 
September  -  October  2004. 
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Figure  21 .  Forecast  Accuracy  (FAC),  USS  Saipan,  RAS  Forecasts,  September 
-  October  2004. 
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Figure  22.  Probability  of  Detection  (POD),  USS  Saipan,  Aviation  Forecasts, 
September  -  October  2004. 
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Figure  23.  Probability  of  Detection  (POD),  USS  Saipan,  LCAC  Forecasts, 
September  -  October  2004. 
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Figure  24.  Probability  of  Detection  (POD),  USS  Saipan,  LCU  Forecasts, 
September  -  October  2004. 
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Figure  25.  Probability  of  Detection  (POD),  USS  Saipan,  Smallboat  Forecasts, 
September  -  October  2004. 
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Figure  26.  Probability  of  Detection  (POD),  USS  Saipan,  RAS  Forecasts, 
September  -  October  2004. 
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Figure  27.  Percent  of  missions  and  weapons  that  were  saved  or  potentially 
saved  as  the  result  of  METOC  support  to  NSAWC. 
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:igure  28.  NSAWC  mission  changes  made  during  initial  team  planning  in 
response  to  weather  forecasts. 
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-igure  29.  Tactical  changes  made  to  NSAWC  missions  in  response  to  weather 
observed  immediately  prior  to  mission  takeoff. 
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Figure  30.  Percent  of  missions  conducted  by  NSAWC  forecasted  to  be 
negatively  impacted  by  specific  weather  phenomena. 
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Figure  31 .  Percent  of  missions  conducted  by  NSAWC  that  were  negatively 
impacted  by  specific  weather  phenomena. 
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1.  EVENTS  IN  THIS  STUDY 

2.  EVENTS  IMPACTED  BY  WEATHER 

3.  PERCENT  OF  EVENTS  IMPACTED  BY  WEATHER 

4.  IMPACT  BY  WEATHER  TYPE 

Excessive  Surface  Wind 
Excessive  Winds  Aloft 
Altitude  Restrictions 
Cloud  Thickness/Layers 
Reduced  Visibility  due  to  Dust 
Reduced  Visibility  to  Fog 
Reduced  Visibility  due  to  Other 

5.  INITIAL  TEAM  PLANNING  CHANGES 
No  changes  required 

Event  Delayed/Rescheduled 
Weapons  change  for  High  Winds 
Weapon  changed  for  visibility  and  or  ceiling 
Other 

■  None 


Delay 
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Wep  Wind 
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■  Other 
6.  FORECASTER  ACCURACY 

No  changes  were  required  during  team  planning. 

Weather  was  forecasted  correctly. 

Change  was  correct  decision 
Weather  was  forecasted  correctly. 

Mission  planner  believed  accurate  forecast  but  made  no  change 
Weather  forecast  was  wrong. 

Change  was  unnecessary.  Forecast  falsely  steered  mission 
plan  to  be  over  conservative. 

Weather  forecast  was  wrong. 

Change  was  required  as  the  weather  impact  was  greater  than  forecasted 
Not  enough  information  to  know  if  decision  was  good  or  bad. 

Other 
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rigure  32.  Example  of  NPMOD  Fallon  on-demand  output  report  generated  by 
using  a  PHP  program  embedded  into  NPS  METOC  Metrics  Support 
Site  web  page. 
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APPENDIX  B:  CODE  FOR  OUTPUT  REPORT  FOR  NPMOD 

FALLON 


The  code  that  was  used  to  create  the  output  report  developed  for 
NPMOD,  Fallon  is  listed  below.  It  can  also  be  viewed  at  the  following  web  link: 
http://wx.met.nps.navv.mil/~mdbutler/fallon/metrics.htm 

< ! DOCTYPE  HTML  PUBLIC  "-//W3C//DTD  HTML  4.01  Transitional//EN" 


"http : / /www . w3 . org/TR/html4/loose . dtd"> 


< !  — 

PROGRAM  TITLE:  airwing_report_graph . php 

- > 

< !  — 

Created  June  19,  2005 

by  LCDR  Mark  Butler 

- > 

< !  — 

Naval  Post-Graduate  School,  Monterey,  CA 

- > 

< !  — 

Program  called  from  airwing_selection . php 

- > 

< !  — 

Data  needed:  $airwing 

_id 

- > 

< !  — 

Requires  connection  to 

database  "fallon" 

- > 

< !  — 

Programs  needed:  1) 

charts . php 

- > 

< !  — 

2) 

charts . swf 

- > 

< !  — 

3) 

report_negimpact . php 

- > 

< !  — 

4) 

report_change . php 

- > 

< !  — 

5) 

results_impact . php 

- > 

<HTML> 

<HEAD> 

<TITLE>NPMOD  FALLON  METRICS  OUTPUT  REPORT</TITLE> 

</HEAD> 

<B0DY> 

< ! — BANNER  AND  NAVIGATION  LINKS — > 

<table  border="6"  bgcolor="#003366"  cellpadding=" 0 "  cellspacing=" 0 " 
style="border-collapse :  collapse"  width="100%"  id="AutoNumberl " 

height="72"> 

<tr> 

<td  width="20%"  bordercolor="#003366"ximg  border="0"  src="nps . bmp" 
></td> 

<td  width="60%"  bordercolor="#003366"><p  align="center " 

style="margin-top : 

0;  margin-bottom:  0"> 

<font  color="#FFFF00"  size="14"  face="Arial"> 

<strong>Strike  Data  Results  </ strongx/ f ont></td> 

<td  width="20%"  bordercolor="#003366"ximg  border="0" 

src=" f  allon_blue  .  bmp"  x/td> 

</tr> 

</table> 

<table  border="3"  bgcolor="#003366"  cellpadding=" 0 "  cellspacing=" 0 " 
style="border-collapse :  collapse"  width="100%"  id="AutoNumberl " 

al ign=" center "> 

<tr> 

<td  width="10%"  bordercolor="#003366"x/td> 

<td  bordercolor="#003366"  align="center "> 

<a  href =" http : / / wx . met . nps . navy . mil/~mdbutler/ index . html " 
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STYLE="text-decoration:  none"><FONT  COLOR="#FFFFOO" 
onMouseOver="this . style . color  =  ' #FFFFFF ' " 

onMouseOut="this . style . color  =  ' #FFFF00 ' "><f ont  size="2" 

face="Arial"> 

Metrics  Home  </font></a> 

</td> 

<td  bordercolor="#003366"  align="center "> 

<a  href ="http : / / wx . met . nps . navy . mil/~mdbutler/ fallon/metrics . htm" 
STYLE="text-decoration :  none"><FONT  COLOR="#FFFFOO" 
onMouseOver="this . style . color  =  ' #FFFFFF ' " 

onMouseOut="this  .  style  .  color  =  '  #FFFF00  '  "xfont  size="2" 

face="Arial"> 

NSAWC  Home  </fontx/a> 

</td> 

<td  bordercolor="#003366"  align="center "> 

<a 

href =" http : / / wx . met . nps . navy . mil/~mdbutler/ f allon/ enter /main . html " 
STYLE="text-decoration  :  none  "XFONT  COLOR="#FFFFOO" 
onMouseOver="this . style . color  =  ' #FFFFFF ' " 

onMouseOut="this  .  style  .  color  =  '  #FFFF00  '  "xfont  size="2" 

face="Arial"> 

NSAWC  Enter</f ontx/a> 

</td> 

<td  bordercolor="#003366"  align="center "> 

<a 

href =" http : / / wx . met . nps . navy . mil/~mdbutler/ fallon/delete/delete . php " 
STYLE="text-decoration  :  none  "XFONT  COLOR="#FFFFOO" 
onMouseOver="this . style . color  =  ' #FFFFFF ' " 

onMouseOut="this  .  style  .  color  =  '  #FFFF00  '  "Xfont  size="2" 

face="Arial"> 

NSAWC  Delete</fontx/a> 

</td> 

<td  bordercolor="#003366"  align="center "> 

<a 

href =" http : / / wx . met . nps . navy . mil/~mdbutler/ f allon/ review/ review . php " 
STYLE="text-decoration:  none  "XFONT  COLOR="#FFFFOO" 
onMouseOver="this . style . color  =  ' #FFFFFF ' " 

onMouseOut="this  .  style  .  color  =  '  #FFFF00  '  "xfont  size="2" 

face="Arial"> 

NSAWC  Review</fontx/a> 

</td> 

<td  bordercolor="#003366"  align="center "> 

<a 

href =" http : / / wx . met . nps . navy . mil/~mdbutler/ f allon/ results/ airwing_selec 
tion . php " 

STYLE="text-decoration:  none  "XFONT  COLOR="#FFFFOO" 
onMouseOver="this . style . color  =  ' #FFFFFF ' " 

onMouseOut="this  .  style  .  color  =  '  #FFFF00  '  "xfont  size="2" 

face="Arial"> 

NSAWC  Results</fontx/a> 

</td> 

<td  bordercolor="#003366"  align="center "> 

<a  href ="http : / / wx . met . nps . navy . mi l/~mdbutler/ help /help . html " 
STYLE="text-decoration  :  none  "XFONT  COLOR="#FFFFOO" 
onMouseOver="this . style . color  =  ' #FFFFFF ' " 

onMouseOut="this  .  style  .  color  =  '  #FFFF00  '  "xfont  size="2" 

face="Arial"> 
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About  Metrics</f ont></a> 

</td> 

<td  width="10%"  bordercolor="#003366"x/td> 

</tr> 

</table> 

<?php 

// 

//  IMPORT  $airwing_id  from  airwing_selection . php 

// 

$airwing_id=$_POST [ ' airwing_id ' ] ; 

// 

//CONNECT  TO  DATABASE 

// 

$host=" localhost " ; 

$user="root" ; 

$password=" " ; 

$database="fallon" ; 

$connection  =  mysql_connect ($host, $user, $password) 
or  die  ("couldn't  connect  to  server"); 

$db  =  mysql_select_db ($database, $connection) 
or  die  ("Couldn't  select  database"); 


/* - 

CHECK  TO  SEE  IF  AIRWING  IS  SPECFIC  AIRWING  OR  ALL  AIRWINGS 
AND  DEFINE  SQL  QUERY  TO  BE  USED  TO  RETRIVE  DATA 

- */ 

if  ( $airwing_id=="ALL" )  { 

$ query  =  "SELECT 

mission_id,  itp_delayed, itp_canceled, itp_schedule_other , 

itp_lowwar, itp_highwar, itp_splitwar , itp_war_other , itp_partial_nostrike, 
itp_partial_nobombs, itp_partial_missed, itp_partial_other , 

mission_surf ace_winds, mission_aloft_winds, mission_altitude, mission_clou 
ds, 

mission_visibility, mission_other , mission_visibility_phenomena, 
mission_altitude_phenomena, mission_thermal, 
changed, change_verif , 

air_f orecast_wind, air_observe_wind, air_f orecast_ceiling, 
air_observe_ceiling, air_f orecast_visibility, air_observe_visibility, 
ir_sensor, ir_out_acc, ir_out_20plus, ir_out_201ess, 

ir_plan_none, ir_plan_poss, ir_plan_thermal, ir_plan_taws, ir_plan_other 
FROM  strikedata"; 

}  else  { 

$ query  =  "SELECT 

mission_id, itp_delayed, itp_canceled,  itp_schedule_other , 

itp_lowwar,  itp_highwar, itp_splitwar , itp_war_other , itp_partial_nostrike, 
itp_partial_nobombs, itp_partial_missed, itp_partial_other , 

mission_surf ace_winds, mission_aloft_winds, mission_altitude, mission_clou 
ds, 

mission_visibility, mission_other , mission_visibility_phenomena, 
mission_altitude_phenomena, mission_thermal, 
changed, change_verif , 
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air_f orecast_wind, air_observe_wind, air_f orecast_ceiling, 
air_observe_ceiling, air_f orecast_visibility, air_observe_visibility, 
ir_sensor, ir_out_acc, ir_out_20plus, ir_out_201ess, 

ir_plan_none, ir_plan_poss, ir_plan_thermal, ir_plan_taws, ir_plan_other 
FROM  strikedata  WHERE  airwing_id=$airwing_id" ; 

} 

$result  =  mysql_query ( $query ) 

or  die  ("Couldn't  execute  query."); 

// 

//INITIALIZE  COUNTERS  TO  BE  USED  TO  DETERMINE  METRICS 

// 

$mission_counter=0 ; 

$impact_yes=0 ; 

$none=0 ; 

$delay_impact=0 ; 

$canx=0 ; 

$sched_other=0 ; 

$lowwar=0 ; 

$highwar=0 ; 

$splitwar=0 ; 

$war_other=0 ; 

$partial_nostrike=0 ; 

$partial_nobombs=0 ; 

$partial_other=0 ; 

$partial_missed=0 ; 

$surf_wind=0 ; 

$aloft_wind=0 ; 

$alt=0 ; 

$vis=0 ; 

$vis_pr=0 ; 

$vis_ds=0 ; 

$vis_haze=0 ; 

$vis_f og=0 ; 

$vis_na=0 ; 

$cloud=0 ; 

$therm=0 ; 

$other_impact=0 ; 

$none_changed=0 ; 

$delay=0 ; 

$wep_wind=0 ; 

$wep_vis=0 ; 

$other_changed=0 ; 

$no3a=0 ; 

$correct=0 ; 

$nochange=0 ; 

$wrong_unnec=0 ; 

$wrong_nec=0 ; 

$unknown=0 ; 

$other_verf =0 ; 

$wind_correct=0 ; 

$wind_greater=0 ; 

$wind_less=0 ; 

$ceil_correct=0 ; 
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$ceil_greater=0 ; 

$ceil_less=0 ; 

$vis_correct=0 ; 

$vis_greater=0 ; 

$vis_less=0 ; 

$atf lir_acc=0 ; 

$atf lir_plus=0 ; 

$atf lir_less=0 ; 

$lantirn_acc=0 ; 

$lantirn_plus=0 ; 

$lantirn_less=0 ; 

$nighthawk_acc=0 ; 

$nighthawk_plus=0 ; 

$nighthawk_less=0 ; 

$ir_none=0 ; 

$ir_poss=0 ; 

$ir_thermal=0 ; 

$ir_taws=0 ; 

$ir_other=0 ; 

// 

//  EXECUTION  OF  WHILE  LOOP  TO  RETRIVE  DATA  AND  TO  PERFORM  METRIC 
ANALYSIS 
// 

while  ($row  =  mysql_f etch_array ( $result ) ) 

{ 

extract ( $row) ; 

// 

//MISSION  COUNTER 
// 

$mission_counter=$mission_counter+l ; 

/* - 

WEATHER  WHICH  RESULTED  IN  IMPACTS  IS  ANALYZED 

1 )  DATA  IS  ASSIGNED  TO  VARIABLES  FROM  QUERY 

2)  VARIABLES  ARE  THEN  CHECK  TO  SEE  WHICH  WEATHER 
PHENOMENA  CAUSED  AN  IMPACT 

- */ 

$mission_surf ace_winds=$row [ ' mission_surf ace_winds ' ] ; 
$mission_aloft_winds=$row [ ' mission_aloft_winds ' ] ; 
$mission_altitude=$row [ ' mission_altitude ' ] ; 

$mission_clouds=$row [ ' mission_clouds ' ] ; 

$mission_visibility=$row [ ' mission_visibility ' ] ; 

$mission_other=$row [ ' mission_other ' ]  ; 

$mission_visibility_phenomena=$row [ ' mission_visibility_phenomena ' ] ; 

$mission_altitude_phenomena=$row [ ' mission_altitude_phenomena ' ] ; 
$mission_thermal=$row [ ' mission_thermal ' ] ; 
if  ( $mission_surf ace_winds=="yes " )  { 

$surf_wind=$surf_windtl ; 

} 

if  ( $mission_aloft_winds=="yes " )  { 

$aloft_wind=$aloft_windtl ; 

} 

if  ($mission_altitude=="yes" )  { 

$alt=$alt+l ; 

} 
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if  ( $mission_clouds=="yes " )  { 

$cloud=$cloud+l ; 

} 

if  ( $mission_visibility=="yes "  )  { 

$vis=$vis+l ; 

if  ($mission_visibility_phenomena=="na" )  { 

$vis_na=$vis_na+l ; 

} 

if  ($mission_visibility_phenomena=="fog" )  { 

$vis_f og=$vis_f og+1 ; 

} 

if  ($mission_visibility_phenomena=="haze" )  { 

$vis_haze=$vis_haze+l ; 

} 

if  ($mission_visibility_phenomena=="duststorm" )  { 

$vis_ds=$vis_ds+l ; 

} 

if  ($mission_visibility_phenomena=="precip" )  { 

$vis_pr=$vis_pr+l ; 


if  ( $mission_thermal=="yes " )  { 

$therm=$therm+l ; 

} 

if  ( $mission_other=="yes "  )  { 

$other_impact=$other_impact  +  l ; 

} 

// 

//ANALYZE  INITIAL  TEAM  PLANNING  CHANGES 

// 

$changed=$row [ ' changed ' ]  ; 
switch  ($changed)  { 
case  "none": 

$none_changed=$none_changed+l ; 
break; 

case  "delayed" : 

$delay=$delay+l ; 
break; 

case  "weaponwind" : 

$wep_wind=$wep_wind+l ; 
break; 

case  "weaponvis" : 

$wep_vis=$wep_vis+l ; 
break; 

case  "other": 

$other_changed=$other_changedtl ; 
break; 


//ANALYZE  FORECAST  ACCURACY 

// 

switch  ($change_verif )  { 

case  "no3a"  : 
$no3a=$no3atl ; 
break; 

case  "correct"  : 
$correct=$correct+l; 
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break ; 

case  "nochange"  : 
$nochange=$nochange+l ; 
break; 

case  "wrong_unnec"  : 
$wrong_unnec=$wrong_unnec+l ; 
break ; 

case  "wrong_nec"  : 
$wrong_nec=$wrong_nec+l ; 
break; 

case  "unknown"  : 
$unknown=$unknown+l ; 
break; 

case  "other"  : 

$other_verf =$other_verf +1 ; 
break; 


// 

//ANALYZE  IMPACTS  RESULTING  FROM  WEATHER 

// 

$itp_delayed=$row [ ' itp_delayed ' ] ; 

$itp_canceled=$row [ ' itp_canceled ' ] ; 
$itp_schedule_other=$row [ ' itp_schedule_other ' ] ; 
$itp_lowwar=$row [ ' itp_lowwar ' ] ; 

$itp_highwar=$row [ ' itp_highwar ' ] ; 

$itp_splitwar=$row [ ' itp_splitwar ' ] ; 
$itp_war_other=$row [ ' itp_war_other ' ] ; 
$itp_partial_nostrike=$row [ ' itp_partial_nostrike ' ] ; 
$itp_partial_nobombs=$row [ ' itp_partial_nobombs ' ] ; 
$itp_partial_missed=$row [ ' itp_partial_missed ' ] ; 
$itp_partial_other=$row [ ' itp_partial_other ' ] ; 
if  ( $itp_delayed=="no" 

$itp_schedule_other=="no" 

and  $itp_lowwar=="no" 

$itp_splitwar=="no" 

and  $itp_war_other=="no" 
$itp_partial_nobombs=="no" 

and  $itp_partial_other=="no 
$none=$none+l ; 

} 

if  ( $itp_delayed=="yes " 

$itp_schedule_other=="yes " 

or  $itp_lowwar=="yes " 

$itp_splitwar=="yes " 

or  $itp_war_other=="yes " 
$itp_partial_nobombs=="yes " 

or  $itp_partial_other=="yes 
$impact_yes=$impact_yes+l ; 

} 

if  ($itp_delayed=="yes" )  { 

$delay_impact=$delay_impact+l ; 

} 

if  ($itp_canceled=="yes" )  { 

$canx=$canx+l ; 

} 

if  ($itp_schedule_other=="yes" )  { 

$sched_other=$sched_other+l ; 


and 

$ itp_canceled== " no " 

and 

and 

$itp_highwar=="no" 

and 

and  $itp_partial_nostrike=="no"  and 
"  and  $itp_partial_missed=="no"  )  { 

or  $itp_canceled=="yes "  or 

or  $itp_highwar=="yes"  or 

or  $itp_partial_nostrike=="yes "  or 
or  $itp_partial_missed=="yes"  )  { 
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} 

if  ($itp_lowwar=="yes" )  { 

$lowwar=$lowwar+l ; 

} 

if  ($itp_highwar=="yes" )  { 

$highwar=$highwar+l ; 

} 

if  ($itp_splitwar=="yes" )  { 

$ sp 1 it war =$ split war +1 ; 

} 

if  ($itp_war_other=="yes" )  { 

$war_other=$war_other+l ; 

} 

if  ($itp_partial_nostrike=="yes" )  { 

$partial_nostrike=$partial_nostrike+l ; 

} 

if  ($itp_partial_nobombs=="yes" )  { 

$partial_nobombs=$partial_nobombs+l ; 

} 

if  ($itp_partial_other=="yes" )  { 

$partial_other=$partial_other+l ; 

} 

if  ($itp_partial_missed=="yes" )  { 

$partial_missed=$partial_missed+l ; 

} 

// 

//ANALYZE  OF  FORECAST  ACCURACY  OF  LOCAL  AIRFIELD  CONDITIONS 

// 

$air_f orecast_wind=$row [ ' air_f orecast_wind ' ] ; 
$air_observe_wind=$row [ ' air_observe_wind ' ] ; 

$air_f orecast_ceiling=$row [ ' air_f orecast_ceiling ' ] ; 
$air_observe_ceiling=$row [ ' air_observe_ceiling ' ] ; 

$air_f orecast_visibility=$row [ ' air_f orecast_visibility ' ] ; 
$air_observe_visibility=$row [ ' air_observe_visibility ' ] ; 
if  ( $air_f orecast_wind==$air_observe_wind)  { 
$wind_correct=$wind_correct+l ; 

} 

if ($air_observe_wind=="0-18"  and  $air_forecast_wind=="19-24"  )  { 

$wind_less=$wind_less+l ; 

} 

if ($air_observe_wind=="0-18"  and  $air_forecast_wind=="25"  )  { 

$wind_less=$wind_less+l ; 

} 

if ($air_observe_wind=="19-24"  and  $air_forecast_wind=="0-18"  )  { 

$wind_greater=$wind_greater+l ; 

} 

if ($air_observe_wind=="19-24"  and  $air_forecast_wind=="25"  )  { 

$wind_less=$wind_less+l ; 

} 

if ($air_observe_wind=="25"  and  $air_forecast_wind=="0-18"  )  { 

$wind_greater=$wind_greater+l ; 

} 

if ($air_observe_wind=="25"  and  $air_forecast_wind=="19-24"  )  { 

$wind_greater=$wind_greater+l ; 

} 


if 


( $air_f orecast_ceiling== 


$air_observe_ceiling) 
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{ 


$ceil_correct=$ceil_correct  +  l ; 

} 

if ($air_observe_ceiling=="0-2000"  and 

$air_f orecast_ceiling=="2000-10000 "  )  { 

$ceil_less=$ceil_less+l ; 

} 

if ($air_observe_ceiling=="0-2000"  and 

$air_forecast_ceiling==" 10000 "  )  { 

$ceil_less=$ceil_less+l ; 

} 

if ($air_observe_ceiling=="0-2000"  and 

$air_f orecast_ceiling=="noceiling"  )  { 

$ceil_less=$ceil_less+l ; 

} 

if ($air_observe_ceiling=="2000-10000"  and 

$air_forecast_ceiling=="0-2000"  )  { 

$ceil_greater=$ceil_greater+l ; 

} 

if ($air_observe_ceiling=="2000-10000"  and 

$air_forecast_ceiling==" 10000 "  )  { 

$ceil_less=$ceil_less+l ; 

} 

if ($air_observe_ceiling=="2000-10000"  and 

$air_f orecast_ceiling=="noceiling"  )  { 

$ceil_less=$ceil_less+l ; 

} 

if ($air_observe_ceiling=="10000"  and  $air_f orecast_ceiling==" 0- 

2000"  )  { 

$ceil_greater=$ceil_greater+l ; 

} 

if ($air_observe_ceiling=="10000"  and  $air_f orecast_ceiling=="2000- 

10000"  )  { 

$ceil_greater=$ceil_greater+l ; 

} 

if ($air_observe_ceiling=="10000"  and 

$air_f orecast_ceiling=="noceiling"  )  { 

$ceil_less=$ceil_less+l ; 

} 

if ( $air_observe_ceiling=="noceiling"  and 

$air_forecast_ceiling=="0-2000"  )  { 

$ceil_less=$ceil_less  +  l ; 

} 

if ( $air_observe_ceiling=="noceiling"  and 

$air_f orecast_ceiling=="2000-10000 "  )  { 

$ceil_less=$ceil_less+l ; 

} 

if ( $air_observe_ceiling=="noceiling"  and 

$air_forecast_ceiling==" 10000 "  )  { 

$ceil_less=$ceil_less+l ; 

} 

if  ( $air_f orecast_visibility==$air_observe_visibility )  { 

$vis_correct=$vis_correct+l ; 

} 

if ($air_observe_visibility=="0-3"  and  $air_f orecast_visibility==" 3 
or  greater"  )  { 

$vis_less=$vis_less+l ; 

} 
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if ( $air_observe_visibility==" 3  or  greater" 

$air_f orecast_visibility==" 0-3 "  )  { 

$vis_greater=$vis_greater+l ; 

} 

// 

//ANALYSIS  OF  TAWS  DATA 

// 

$ir_sensor=$row [ ' ir_sensor ' ] ; 

$ir_out_acc=$row [ ' ir_out_acc ' ] ; 

$ir_out_20plus=$row [ ' ir_out_20plus ' ]  ; 

$ir_out_201ess=$row [ ' ir_out_201ess ' ] ; 
if  ($ir_sensor=="atf lir" )  { 

if  ($ir_out_acc=="yes" )  { 

$atf lir_acc=$atf lir_acc+l ; 

} 

if  ($ir_out_20plus=="yes" )  { 

$atf lir_plus=$atf lir_plus+l ; 

} 

if  ($ir_out_201ess=="yes" )  { 

$atf lir_less=$atf lir_less+l ; 

} 

} 

if  ($ir_sensor=="lantirn" )  { 

if  ($ir_out_acc=="yes" )  { 

$lantirn_acc=$lantirn_acc+l ; 

} 

if  ($ir_out_20plus=="yes" )  { 

$lantirn_plus=$lantirn_plus+l ; 

} 

if  ($ir_out_201ess=="yes" )  { 

$lantirn_less=$lantirn_less+l ; 

} 

} 

if  ($ir_sensor=="nighthawk" )  { 

if  ($ir_out_acc=="yes" )  { 

$nighthawk_acc=$nighthawk_acc+l ; 

} 

if  ($ir_out_20plus=="yes" )  { 

$nighthawk_plus=$nighthawk_plus+l ; 

} 

if  ($ir_out_201ess=="yes" )  { 

$nighthawk_less=$nighthawk_less+l ; 


$ir_plan_none=$row [ ' ir_plan_none ' ] ; 
$ir_plan_poss=$row [ ' ir_plan_poss ' ] ; 
$ir_plan_thermal=$row [ ' ir_plan_thermal ' ] ; 
$ir_plan_taws=$row [ ' ir_plan_taws ' ] ; 
$ir_plan_other=$row [ ' ir_plan_other ' ] ; 
if  ($ir_plan_none=="yes" )  { 

$ir_none=$ir_none+l ; 

} 

if  ($ir_plan_poss=="yes" )  { 

$ir_poss=$ir_none+l ; 

} 

if  ($ir_plan_thermal=="yes" )  { 


and 
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$ir_thermal=$ir_none+l ; 

} 

if  ($ir_plan_taws=="yes" )  { 

$ir_taws=$ir_none+l ; 

} 

if  ($ir_plan_other=="yes" )  { 

$ir_other=$ir_none  +  l ; 


// 

//DETERMINE  THE  PERCENT  OF  MISSIONS  IMPACTED 

// 

$missions_impacted=round ($impact_yes/ $mission_counter*100 ) ; 

/* - 

IMPACT  BY  WEATHER  TYPE  CALCULATIONS  (QUESTION  #4) 
- */ 

$total_wx_impacts=$surf_wind+$aloft_wind+$alt+$cloud+$vis_ds+$vis_f og+$ 
vis_na+$vis_haze+$vis_pr ; 

$vis_other=$vis_na+$vis_haze+$vis_pr ; 

// 

//calculate  impact  per  mission 

// 

$per_mission_surf_wind=round ($surf_wind/ $total_wx_impacts*100) ; 
$per_mission_aloft_wind=round ( $aloft_wind/ $total_wx_impacts* 100 ) ; 
$per_mission_alt=round ($alt/ $total_wx_impacts*100 ) ; 
$per_mission_cloud=round ($cloud/ $total_wx_impacts*100) ; 
$per_mission_vis=round ($vis/ $total_wx_impacts*100 ) ; 
$per_mission_vis_other=round ($vis_other/ $total_wx_impacts*100) ; 
$per_mission_vis_ds=round ($vis_ds/ $total_wx_impacts*100) ; 
$per_mission_vis_f og=round ($vis_fog/ $total_wx_impacts*100) ; 
$per_mission_therm=round ($therm/ $total_wx_impacts*100) ; 
$per_mission_other_impact=round ( $other_impact/ $total_wx_impacts* 100 ) ; 

/* - 

INITIAL  TEAM  PLANNING  CHANGES  CALCULATIONS  (QUESTION  #5) 

- */ 

$total_changes=$none_changed+$delay+$wep_wind+$wep_vis+$other_changed; 

// 

//calculate  changes  per  mission 

// 

$per_none_changed=round ( $none_changed/ $total_changes*100) ; 
$per_delay=round ($delay/$total_changes*100) ; 

$per_wep_wind=round ($wep_wind/ $total_changes*100 ) ; 

$per_wep_vis=round ( $wep_vis/$total_changes*100 ) ; 
$per_other_changed=round ( $other_changed/ $total_changes*100 ) ; 

/* - 

FORECAST  ACCURACY  CALCULATIONS  (QUESTION  #6) 
- */ 

$tot a 1=$ cor rect+$ nochange +$wrong_unnec+$wrong_nec+$unknown+$other_verf + 
$no3a; 

$per_no3a=round ($no3a/$total*100)  ; 

$per_correct=round ($correct/$total*100)  ; 

$per_nochange=round ($nochange/$total*100) ; 

$per_wrong_unnec=round ( $wrong_unnec/ $total*100) ; 

$per_wrong_nec=round ($wrong_nec/ $total*100) ; 

73 


$per_unknown=round ($unknown/ $total*100) ; 

$per_other_verf =round ( $other_verf / $total*100) ; 

/* - 

WEATHER  IMPACTS  CALCULATIONS  (QUESTION  #7) 
- */ 

$per_mission_none=round ($none/ $mission_counter*100 )  ; 
$per_mission_delay=round ( $delay_impact/ $mission_counter*100) ; 
$per_mission_canx=round ($canx/ $mission_counter*100 ) ; 
$per_mission_sched_other=round ( $sched_other/ $mission_counter *  100 )  ; 
$per_mission_lowwar=round ($lowwar/ $mission_counter*100 )  ; 
$per_mission_highwar=round ($highwar/ $mission_counter*100) ; 
$per_mission_splitwar=round ($splitwar/ $mission_counter*100 ) ; 
$per_mission_war_other=round ($war_other/ $mission_counter*100) ; 
$per_mission_partial_nostrike=round ( $part i a l_no strike/ $mission_counter* 
100)  ; 

$per_mission_partial_nobombs=round ( $partial_nobombs/ $mission_counter* 10 

0)  ; 

$per_mission_partial_missed=round ( $partial_missed/ $mission_counter*100 ) 

r 

$per_mission_partial_other=round ( $partial_other/ $mission_counter* 100 ) ; 

/* - 

AIRFIELD  FORECAST  CALCULATIONS  (QUESTION  #8) 

- */ 

$per_wind_correct=round ( $wind_correct/ $mission_counter*100 )  ; 
$per_wind_less=round ($wind_less/ $mission_counter*100) ; 
$per_wind_greater=round ( $wind_greater/ $mission_counter*100) ; 
$per_ceil_correct=round ( $ceil_correct/ $mission_counter*100 )  ; 
$per_ceil_less=round ($ceil_less/ $mission_counter*100) ; 
$per_ceil_greater=round ( $ceil_greater/ $mission_counter*100) ; 
$per_vis_correct=round ( $vis_correct/ $mission_counter*100) ; 
$per_vis_less=round ($vis_less/ $mission_counter*100 ) ; 
$per_vis_greater=round ( $vis_greater/ $mission_counter*100 ) ; 

/* - 

TAW'S  PERFORMANCE  CALCULATIONS  (QUESTION  #9) 

- */ 

$total_atf lir=$atf lir_acc+$atf lir_plus+$atf lir_less ; 

// 

//use  if  else  statement  to  calculate  percentages 

// 

if  ( $total_atf lir==" 0 " )  { 

$per_atf lir_acc=0 ; 

$per_atf lir_plus=0 ; 

$per_atf lir_less=0 ; 

}  else  { 

$per_atf lir_acc=$atf lir_acc/ $mission_counter* 100 ; 

$per_atf lir_acc=round ($per_atf lir_acc) ; 

$per_atf lir_plus=$atf lir_plus/ $mission_counter* 100 ; 

$per_atf lir_plus=round ( $per_atf lir_plus ) ; 

$per_atf lir_less=$atf lir_less/ $mission_counter*100 ; 

$per_atf lir_less=round ( $per_atf lir_less ) ; 

} 

//use  if  else  statement  to  calculate  percentages  to  ensure 
//no  division  by  zero 
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$total_lantirn=$lantirn_acc+$lantirn_plus+$lantirn_less ; 
if  ( $total_lantirn==" 0 " )  { 

$per_lantirn_acc=0 ; 

$per_lantirn_plus=0 ; 

$per_lantirn_less=0 ; 

}  else  { 

$per_lantirn_acc=$lantirn_acc/ $mission_counter* 100 ; 
$per_lantirn_acc=round ($per_lantirn_acc)  ; 
$per_lantirn_plus=$lantirn_plus/ $mission_counter *  100 ; 
$per_lantirn_plus=round ( $per_lantirn_plus ) ; 
$per_lantirn_less=$lantirn_less/ $mission_counter* 100 ; 
$per_lantirn_less=round ( $per_lantirn_less ) ; 


} 

//use  if  else  statement  to  calculate  percentages  to  ensure 
//no  division  by  zero 

$total_nighthawk=$nighthawk_acc+$nighthawk_plus  +  $nighthawk_less ; 
if  ( $total_nighthawk==" 0 " )  { 

$per_nighthawk_acc=0 ; 

$per_nighthawk_plus=0 ; 

$per_nighthawk_less=0 ; 

}  else  { 

$per_nighthawk_acc=$nighthawk_acc/ $mission_counter* 100 ; 
$per_nighthawk_acc=round ($per_nighthawk_acc) ; 
$per_nighthawk_plus=$nighthawk_plus/ $mission_counter *  100 ; 
$per_nighthawk_plus=round ( $per_nighthawk_plus ) ; 
$per_nighthawk_less=$nighthawk_less/ $mission_counter* 100 ; 
$per_nighthawk_less=round ( $per_nighthawk_less ) ; 

} 

/*  TO  DISPLAY  RESULTS  THE  HTML  IS  EMBEDDED  WITH  ECHO  STATEMENTS 
INTO  THE  PHP  TO  ALLOW  DIRECT  DISPLAY  OF  THE  GRAPHICS  AND  VARIABLES*/ 
// 

//DISPLAY  OF  AIRWING  ID  AND  PRIMARY  METRICS 

// 

echo"  <font  f ace= ' Arial ' ><br> 

<table  align= ' center ' > 

<tr> 

<td><b>METRIC  REPORT  FOR  AIRWING  -  $airwing_id  </b>  </td> 

</tr> 

</table> 

<br> 

<table  width='80%'  > 

<tr> 

<td  width= ' 55% ' ><b>l .  EVENTS  IN  THIS  STUDY  </b></td> 

<td>  $mission_counter</td> 

</tr> 

<tr> 

<td  width= ' 55% ' ><b>2 .  EVENTS  IMPACTED  BY  WEATHER</b></td> 

<td>  $impact_yes</td> 

</tr> 

<tr> 

<td  width= ' 55% ' ><b>3 .  PERCENT  OF  EVENTS  IMPACTED 
WEATHER</b></td> 

<td>  $missions_impacted  </td> 

</tr> 

</table>" ; 


/* - 

DISPLAYS  QUESTION  4:  IMPACT  BY  WEATHER  TYPE 
- */ 


echo" 

<br> 

<table  width= ' 100% ' > 

<tr> 

<B>4 .  IMPACT  BY  WEATHER  TYPE</BR> 

</tr> 

<tr> 

<td  width= ' 40% ' >Excessive  Surface  Wind</td> 

<td>$surf_wind</td> 

<td>$per_mission_surf_wind  %</td> 

</tr> 

<tr> 

<td  width= ' 40% ' >Excessive  Winds  Aloft</td> 

<td>$aloft_wind</td> 

<td>$per_mission_aloft_wind  %</td> 

</tr> 

<tr> 

<td  width= ' 40% ' >Altitude  Restrictions</td> 

<td>$alt</td> 

<td>$per_mission_alt  %</td> 

</tr> 

<tr> 

<td  width= ' 40% ' >Cloud  Thickness/Layers</td> 

<td>$cloud</td> 

<td>$per_mission_cloud  %</td> 

</tr> 

<tr> 

<td  width= ' 40% ' >Reduced  Visibility  due  to  Dust</td> 
<td>$vis_ds</td> 

<td>$per_mission_vis_ds%</td> 

</tr> 

<tr> 

<td  width= ' 40% ' >Reduced  Visibility  to  Fog</td> 

<td>$vis_f og</td> 

<td>$per_mission_vis_f og%</td> 

</tr> 

<tr> 

<td  width= ' 40% ' >Reduced  Visibility  due  to  Other</td> 
<td>$vis_other</td> 

<td>$per_mission_vis_other%</td> 

</tr> 

</table> 

II  . 
r 

// 

//INSERTS  GRAPH  INTO  WEB  PAGE  USING  CHARTS . PHP  AND  CHARTS . SWF  PROGRAMS 

// 

include  "charts . php" ; 

// 

//POSITIONS  CHART 

// 


echo"<table  width='80%'> 
<tr> 

<td> " ; 
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charts . swf 


echo  InsertChart  ( 

"report_negimpact . php?surf_wind=$per_mission_surf_wind 
&aloft_wind=$per_mission_aloft_wind&alt=$per_mission_alt&cloud=$per_mis 
sion_cloud 

&vis_ds=$per_mission_vis_ds&vis_f og=$per_mission_vis_f og&vis_other=$per 
_mission_vis_other " ,  400 , 400 ,  "FFFFFF" )  ; 

/* - 

DISPLAYS  QUESTION  5:  INTIAL  TEAM  PLANNING  CHANGES 

- */ 

echo"</td> 

</tr> 

</table> 

<br> 

<table  width= ' 100%  '  > 

<tr> 

<B>5 .  INITIAL  TEAM  PLANNING  CHANGES </BR> 

</tr> 

<tr> 

<td  width= ' 40% ' >No  changes  required  </td> 

<td>$none_changed</td> 

<td>$per_none_changed  %</td> 

</tr> 

<tr> 

<td  width= ' 40% ' >Event  Delayed/Rescheduled</td> 

<td>$delay</td> 

<td>$per_delay  %</td> 

</tr> 

<tr> 

<td  width= ' 40% ' >Weapons  change  for  High  Winds</td> 
<td>$wep_wind</td> 

<td>$per_wep_wind  %</td> 

</tr> 

<tr> 

<td  width= ' 40% ' >Weapon  changed  for  visibility  and  or  ceiling</td> 
<td>$wep_vis</td> 

<td>$per_wep_vis  %</td> 

</tr> 

<tr> 

<td  width= ' 40% ' >Other  </td> 

<td>$other_changed</td> 

<td>$per_other_changed%</td> 

</tr> 

</table>" ; 

echo"<table  width='80%'> 

<tr> 

<td>" ; 

echo  InsertChart  (  "charts. swf" 

"report_change . php?none_changed=$none_changed 
&delay=$delay&wep_wind=$wep_wind&wep_vis=$wep_vis 
&other_changed=$other_changed" ,  400 , 400 ,  "FFFFFF" ) ; 

/* - 

DISPLAYS  QUESTION  6:  FORECAST  ACCURACY 

- */ 

echo"</td> 

</tr> 

</table> 


ctable  width= ' 100% ' > 

<tr> 

<B>6 .  FORECASTER  ACCURACY  </BR> 

</tr> 

<tr> 

<td  width= ' 50% ' >No  changes  were  required  during  team  planning. 
</td> 

<td>$no3a</td> 

<td>$per_no3a  %</td> 

</tr> 

<tr> 

<td  width= ' 50% ' >Weather  was  forecasted 

correctly . <br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 

<font  size='-l'>  Change  was  correct  decision</f ont></td> 
<td>$correct</td> 

<td>$per_correct  %</td> 

</tr> 

<tr> 

<td  width= ' 50% ' >Weather  was  forecasted 

correctly . <br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 

<font  size= ' -1 ' >Mission  planner  believed  accurate  forecast  but 
made  no  change</f ont></td> 

<td>$nochange</td> 

<td>$per_nochange  %</td> 

</tr> 

<tr> 

<td  width= ' 50% ' >Weather  forecast  was 

wrong . <br>&nbsp;  &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 

<font  size= ' -1 ' >Change  was  unnecessary.  Forecast  falsely  steered 
mission<br> 

&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;  plan  to  be  over  conservative. 
</td> 

</ f ont><td>$wrong_unnec</td> 

<td>$per_wrong_unnec  %</td> 

</tr> 

<tr> 

<td  width= ' 50% ' >Weather  forecast 

<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 

<font  size= ' -1 ' >Change  was  required  as  the 
greater  than  forecasted  </td> 

</ f ont><td>$wrong_nec</td> 

<td>$per_wrong_nec%</td> 

</tr> 

<tr> 

<td  width= ' 50% ' >Not  enough  information  to  know  if  decision  was  good 
or  bad.</td> 

<td>$unknown</td> 

<td>$per_unknown%</td> 

</tr> 

<tr> 

<td  width= ' 50% ' >Other  </td> 

<td>$other_verf </td> 

<td>$per_other_ver f %</td> 

</tr> 

</table> 

<table  width='80%'> 

<tr> 


was  wrong, 

weather  impact  was 


78 


<td>" ; 

/* - 

DISPLAYS  QUESTION  7:  WEATHER  IMPACTS 
- */ 

echo  InsertChart  (  "charts . swf" ,  "results_change . php?per_no3a=$per_no3a 
&per_correct=$per_correct&per_nochange=$per_nochange&per_wrong_unnec=$p 
er_wrong_unnec 

&per_wrong_nec=$per_wrong_nec&per_unknown=$per_unknown&per_other=$per_o 

ther_verf ",  900, 400,  "FFFFFF" )  ; 

echo"</td> 

</tr> 

</table> 

<table  width= ' 100% ' > 

<tr> 

<B>7 .  WEATHER  IMPACTS  </BR> 

</tr> 

<tr> 

<td  width= ' 40% ' >No  Impacts</td> 

<td>$none</td> 

<td>$per_mission_none  %</td> 

</tr> 

<tr> 

<tr> 

<td  width= ' 40% ' >Total  Event  Delayed/Rescheduled  </td> 
<td>$delay_impact</td> 

<td>$per_mission_delay  %</td> 

</tr> 

<td  width= ' 40% ' >Entire  event  canceled  </td> 

<td>$canx</td> 

<td>$per_mission_canx  %</td> 

</tr> 

<tr> 

<td  width= ' 40% ' >Low  war  </td> 

<td>$lowwar</td> 

<td>$per_mission_lowwar  %</td> 

</tr> 

<tr> 

<td  width= ' 40% ' >High  war  </td> 

<td>$highwar</td> 

<td>$per_mission_highwar  %</td> 

</tr> 

<tr> 

<td  width= ' 40% ' >Split  war  </td> 

<td>$splitwar</td> 

<td>$per_mission_splitwar%</td> 

</tr> 

<tr> 

<td  width= ' 40% ' >Non  strike  aircraft  did  not  complete  mission  </td> 
<td>$partial_nostrike</td> 

<td>$per_mission_partial_nostrike%</td> 

</tr> 

<tr> 

<td  width= ' 40% ' >Bomber  did  not  drop  </td> 

<td>$partial_nobombs</td> 

<td>$per_mission_partial_nobombs%</td> 

</tr> 

<tr> 
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<td  width= ' 40% ' >Bombs  missed  target  </td> 

<td>$partial_missed</td> 

<td>$per_mission_partial_missed%</td> 

</tr> 

<tr> 

<td  width= ' 40% ' >Other  partial  mission  </td> 

<td>$partial_other</td> 

<td>$per_mission_partial_other%</td> 

</tr> 

</table>" ; 

echo"<table  width='100%'  align= ' center ' > 

<tr> 

<td>" ; 

echo  InsertChart  (  "charts . swf" , 

"results_impact . php?delay=$per_mission_delay 

&canx=$per_mission_canx&lowwar=$per_mission_lowwar 

&highwar=$per_mission_highwar&splitwar=$per_mission_splitwar 

&partial_nostrike=$per_mission_partial_nostrike&partial_nobombs=$per_mi 

ssion_partial_nobombs 

&partial_missed=$per_mission_partial_missed&partial_other=$per_mission_ 

partial_other 

&none=$per_mission_none" , 600 , 400 , "FFFFFF" ) ; 

/* - 

DISPLAYS  QUESTION  8:  FORECASTER  VERIFICATION 

- */ 

echo"</td> 

</tr> 

</table> 

<table  width= ' 100% ' > 

<tr> 

<B>8 .  FORECASTER  VERIFICATION  </BR> 

</tr> 

<tr> 

<td  width= ' 30% ' >Winds  =  Forecast</td> 

<td>$wind_correct</td> 

<td>$per_wind_correct  %</td> 

</tr> 

<tr> 

<tr> 

<td  width= ' 30% ' >Winds  >  Forecast  </td> 

<td>$wind_greater</td> 

<td>$per_wind_greater  %</td> 

</tr> 

<td  width= ' 30% ' >Winds  <  Forecast  </td> 

<td>$wind_less</td> 

<td>$per_wind_less  %</td> 

</tr> 

<tr> 

<td  width= ' 30% ' ></td> 

<td></td> 

<td></td> 

</tr> 

<tr> 

<td  width= ' 30% ' >Ceiling  =  Forecast</td> 

<td>$ceil_correct</td> 

<td>$per_ceil_correct  %</td> 

</tr> 
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<tr> 

<tr> 

<td  width= ' 30% ' >Ceiling  >  Forecast  </td> 
<td>$ceil_greater</td> 

<td>$per_ceil_greater  %</td> 

</tr> 

<td  width= ' 30% ' >Ceiling  <  Forecast  </td> 
<td>$ceil_less</td> 

<td>$per_ceil_less  %</td> 

</tr> 

<tr> 

<td  width= ' 30% ' ></td> 

<td></td> 

<td></td> 

</tr> 

<tr> 

<td  width= ' 30% ' >Visibility  =  Forecast</td> 
<td>$vis_correct</td> 

<td>$per_vis_correct  %</td> 

</tr> 

<tr> 

<tr> 

<td  width= ' 30% ' >Visibility  >  Forecast  </td> 
<td>$vis_greater</td> 

<td>$per_vis_greater  %</td> 

</tr> 

<td  width= ' 30% ' >Visibility  <  Forecast  </td> 

<td>$vis_less</td> 

<td>$per_vis_less  %</td> 

</tr> 

</table><br> 

<table  width= ' 100% ' > 

<tr> 

<B>9a .  TAWS  ACCURACY  (AT  FLIR) </B> 

</tr> 

<tr> 

<td  width= ' 30% ' >WOF  =  TAWS  Prediction</td> 

<td>$atf lir_acc</td> 

<td>$per_atf lir_acc  %</td> 

</tr> 

<tr> 

<tr> 

<td  width= ' 30% ' >WOF  >  TAWS  Prediction  </td> 

<td>$atf lir_plus</td> 

<td>$per_atf lir_plus  %</td> 

</tr> 

<td  width= ' 30% ' >WOF  <  TAWS  Prediction  </td> 

<td>$atf lir_less</td> 

<td>$per_atf lir_less  %</td> 

</tr><br> 

<tr> 

<td  width=  '  30%  '  ><b>9b  .  TAWS  ACCURACY  (LANTRN)  </bx/td> 
<tdx/td> 

<tdx/td> 

</tr> 

<trx/tr> 

<tr> 


81 


<td  width= ' 30% ' >WOF  =  TAWS  Prediction</td> 

<td>$lantirn_acc</td> 

<td>$per_lantirn_acc  %</td> 

</tr> 

<tr> 

<tr> 

<td  width= ' 30% ' >WOF  >  TAWS  Prediction  </td> 

<td>$lantirn_plus</td> 

<td>$per_lantirn_plus  %</td> 

</tr> 

<td  width= ' 30% ' >WOF  <  TAWS  Prediction  </td> 

<td>$lantirn_less</td> 

<td>$per_lantirn_less  %</td> 

</tr> 

<tr> 

<td  width= ' 30% ' ><b>9c .  TAWS  ACCURACY  (NIGHT  HAWK) </b></td> 

<td></td> 

<td></td> 

</tr> 

<tr> 

<td  width= ' 30% ' >WOF  =  TAWS  Prediction</td> 

<td>$nighthawk_acc</td> 

<td>$per_nighthawk_acc  %</td> 

</tr> 

<tr> 

<tr> 

<td  width= ' 30% ' >WOF  >  TAWS  Prediction  </td> 

<td>$nighthawk_plus</td> 

<td>$per_nighthawk_plus  %</td> 

</tr> 

<td  width= ' 30% ' >WOF  <  TAWS  Prediction  </td> 

<td>$nighthawk_less</td> 

<td>$per_nighthawk_less  %</td> 

</tr> 

</table><br> 

<table  width= ' 100% ' > 

<tr> 

<B>10 .  TAWS  IMPACT  TO  MISSION  PLANNING  </BR> 

</tr> 

<tr> 

<td  width= ' 40% ' >No  changes 

required. <br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 

<font  size='-l'>  There  is  no  forecast  impact  to  the  sensor. 
</f  ontx/td> 

<td>$ir_none</td> 

</tr> 

<tr> 

<td  width= ' 4 0% ' >No  changes  required. 

<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 

<font  size= ' ~1 ' >Possible  IR  Sensor  impact  for  visibility/ceiling 
noted  earlier .  </fontx/td> 

<td>$ir_poss</td> 

</tr> 

<tr> 

<td  width= ' 40% ' >Mission  change  for  thermal  crossover . </td> 
<td>$ir_thermal</td> 
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</tr> 

<tr> 

<td  width= ' 4 0% ' >Attack  heading  change  per  TAWS  ranges  prediction. 
</td> 

<td>$ir_taws</td> 

</tr> 

<tr> 

<td  width= ' 40% ' >Other</td> 

<td>$ir_other</td> 

</tr> 

</table> " ; 

?> 

< ! — DISPLAY  CONTACT  INFORMATION  AT  BOTTOM  OF  PAGE  — > 
<br><br><br><br><br><brxbr> 

<p  align="center ">Points  of  Contact :<br> 

&nbsp; &nbsp; &nbsp; Tom  Murphree,  Ph.D.  <i>murphree@nps . edu</ixbr> 

&nbsp;  &nbsp;  &nbsp;  Mark  Butler,  LCDR,  USN  <i>mdbutler@nps  .  edu</ixbr> 
<br><brx/p> 

<a  href = ' http : //wx . met . nps . navy . mil /~mdbutler/ fallon/metrics . htm ' > 

Return  to  Main  Menu</a> 

</BODY> 

</HTML> 
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1.  Defense  Technical  Information  Center 
Ft.  Belvoir,  Virginia 

2.  Dudley  Knox  Library 
Naval  Postgraduate  School 
Monterey,  California 

3.  Professor  Tom  Murphree 
Naval  Postgraduate  School 
Monterey,  California 

4.  Professor  Carlyle  H.  Wash 
Naval  Postgraduate  School 
Monterey,  California 

5.  CDR  Steve  Woll 
Organization  of  Addressee 
City,  State 

6.  LCDR  Alex  Cantu 
NPMOD  Fallon 
Fallon,  Nevada 

7.  Capt  Jeff  Jarry 

HQ  Air  Force  Weather  Agency 
Omaha,  Nebraska 

8.  LCDR  Jake  Hinz 
USS  Nimitz 

San  Diego,  California 

9.  Sharon  Prichard 

Naval  Postgraduate  School 
Monterey,  California 
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